Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.25163072 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.015657796 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.015075111 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1203 1204 1205 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.866845601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.883900704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885005987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885104962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885117928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885141028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"Crucible a98e4894-9e03-43fa-90d6-1934dfdbb2c9 has session id: ad1276dc-cc58-4ac5-b658-099771a638bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885718711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Upstairs opts: Upstairs UUID: a98e4894-9e03-43fa-90d6-1934dfdbb2c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.885789393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Crucible stats registered with UUID: a98e4894-9e03-43fa-90d6-1934dfdbb2c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.88580503Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:40.885881215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886473425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.8865288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886548579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886746809Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"task":"crutest"} {"msg":"a98e4894-9e03-43fa-90d6-1934dfdbb2c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886794651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886808827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886819712Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886833699Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886847915Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886861631Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886875307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.886933604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:36773)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.88703443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:33849)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.887051628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:61317)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.887066925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.888966249Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.888994912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889067295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889806081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889844999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889918683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.88993545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889951768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889966965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.889981891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890016137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890031594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890046341Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890062448Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890075974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890089861Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890103567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890120084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"":"downstairs","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890134601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890147107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"0","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890161273Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"1","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890193898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"client":"2","":"downstairs","session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890208705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890222141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"a98e4894-9e03-43fa-90d6-1934dfdbb2c9 is now active with session: ad1276dc-cc58-4ac5-b658-099771a638bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890236727Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890249153Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208,"session_id":"ad1276dc-cc58-4ac5-b658-099771a638bd"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890262439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} {"msg":"Activated sub_volume a98e4894-9e03-43fa-90d6-1934dfdbb2c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:40.890273954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:44.960256255Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1208} Begin loop 1 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.987545025Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.999287259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.999608206Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.999645482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.999657768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:44.999675486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"Crucible 7f1526c2-a2a1-4848-b432-d3d363e5a55d has session id: 17e65cab-8cba-41b4-b0f4-b804493b624a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000087073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Upstairs opts: Upstairs UUID: 7f1526c2-a2a1-4848-b432-d3d363e5a55d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000121369Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Crucible stats registered with UUID: 7f1526c2-a2a1-4848-b432-d3d363e5a55d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000137356Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:45.000151103Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000312316Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000347552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000363159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000448738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000464155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"7f1526c2-a2a1-4848-b432-d3d363e5a55d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000485615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000498331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000512357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000526413Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"2","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00054011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000554616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000569143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"ds_connection connected from Ok(127.0.0.1:48850)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000640906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45153)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00076094Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"ds_connection connected from Ok(127.0.0.1:65054)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.000788223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001361413Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001389186Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001406344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"2","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001724139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001772961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001830817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001846984Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001863562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001878659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001893616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001909803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00192438Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001939056Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001955384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00196878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.001989029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002002285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002015671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"":"downstairs","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002031368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002043974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"0","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00205761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"1","":"downstairs","session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002071397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002088384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.00210178Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"7f1526c2-a2a1-4848-b432-d3d363e5a55d is now active with session: 17e65cab-8cba-41b4-b0f4-b804493b624a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002116797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002129443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210,"session_id":"17e65cab-8cba-41b4-b0f4-b804493b624a"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002142539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} {"msg":"Activated sub_volume 7f1526c2-a2a1-4848-b432-d3d363e5a55d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:45.002154124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 110 len 24576 data: 2 2 2 2 2 2 02/30 Write block 323 len 32768 data: 2 2 2 2 2 2 2 2 03/30 Write block 134 len 28672 data: 2 2 2 2 2 2 2 04/30 Flush 05/30 Write block 266 len 32768 data: 2 2 2 2 2 2 2 2 06/30 Write block 324 len 28672 data: 3 3 3 3 3 3 3 07/30 Write block 180 len 24576 data: 2 2 2 2 2 2 08/30 Read block 101 len 28672 09/30 Read block 588 len 8192 10/30 Write block 172 len 12288 data: 2 2 2 11/30 Write block 282 len 32768 data: 2 2 2 2 2 2 2 2 12/30 Write block 170 len 36864 data: 2 2 3 3 3 2 2 2 2 13/30 Read block 338 len 12288 14/30 Flush 15/30 Write block 176 len 4096 data: 3 16/30 Read block 165 len 28672 17/30 Write block 244 len 16384 data: 2 2 2 2 18/30 Read block 569 len 20480 19/30 Flush 20/30 Write block 254 len 32768 data: 2 2 2 2 2 2 2 2 21/30 Read block 353 len 40960 22/30 Read block 503 len 12288 23/30 Read block 271 len 36864 24/30 Read block 536 len 12288 25/30 Flush 26/30 Write block 51 len 16384 data: 2 2 2 2 27/30 Read block 372 len 28672 28/30 Read block 129 len 32768 29/30 Read block 360 len 20480 30/30 Read block 350 len 4096 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 8 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:45.014190653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1210} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 T T F <--- 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 1 2 2 1 F F T <--- 6 120-139 2 2 1 2 2 1 F F F <--- 7 140-159 2 2 1 2 2 1 F F F <--- 8 160-179 2 2 1 4 4 1 F F F <--- 9 180-199 2 2 1 3 3 1 F F F <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 1 5 5 1 F F F <--- 13 260-279 2 2 1 5 5 1 F F F <--- 14 280-299 2 2 1 3 3 1 F F F <--- 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 1 3 3 1 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 5 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 3 3 3 6 6 6 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 2 2 2 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 2 2 2 2 2 2 F F F 8 160-179 2 2 2 4 4 4 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 2 2 2 5 5 5 F F F 14 280-299 2 2 2 3 3 3 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 3 3 3 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 6 Begin loop 2 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.487025778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.498606469Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.498926064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.498967133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.498979749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499007661Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"Crucible 91f666f1-3d5a-41c3-9a44-85ae40404a5c has session id: fc844bc6-1601-4b27-9145-fc3a9baf8a86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499314451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Upstairs opts: Upstairs UUID: 91f666f1-3d5a-41c3-9a44-85ae40404a5c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499346816Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Crucible stats registered with UUID: 91f666f1-3d5a-41c3-9a44-85ae40404a5c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499361662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:49.499374808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499528278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499551779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499566095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499693273Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499711442Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"91f666f1-3d5a-41c3-9a44-85ae40404a5c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499732371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499745077Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499759253Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.49977326Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499787126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499822022Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499847323Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"ds_connection connected from Ok(127.0.0.1:46543)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499866522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"ds_connection connected from Ok(127.0.0.1:58401)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499882039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"ds_connection connected from Ok(127.0.0.1:34308)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.499897426Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.500669077Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.500707945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.500724953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.50104764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501065958Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 6, 1, 1, 2, 2, 2, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.50109242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[0]R generation[0..12]: [1, 1, 3, 1, 1, 2, 2, 2, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501108347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501124665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 6, 1, 1, 2, 2, 2, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501139632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[1]R generation[0..12]: [1, 1, 3, 1, 1, 2, 2, 2, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501154418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501170476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 6, 1, 1, 2, 2, 2, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501188384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[2]R generation[0..12]: [1, 1, 3, 1, 1, 2, 2, 2, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501203211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501219428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501233064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501247171Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Next flush: 7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501266529Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501279905Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"":"downstairs","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501293942Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501306488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"0","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501320554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"1","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.50133452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"client":"2","":"downstairs","session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501348877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501362183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"91f666f1-3d5a-41c3-9a44-85ae40404a5c is now active with session: fc844bc6-1601-4b27-9145-fc3a9baf8a86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501380871Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501393567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218,"session_id":"fc844bc6-1601-4b27-9145-fc3a9baf8a86"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501406783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} {"msg":"Activated sub_volume 91f666f1-3d5a-41c3-9a44-85ae40404a5c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:49.501420079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 114 len 36864 data: 3 3 2 2 2 2 2 2 2 03/30 Write block 143 len 16384 data: 2 2 2 2 04/30 Write block 286 len 4096 data: 3 05/30 Write block 108 len 4096 data: 2 06/30 Read block 313 len 40960 07/30 Read block 376 len 4096 08/30 Write block 33 len 16384 data: 2 2 2 2 09/30 Read block 565 len 4096 10/30 Read block 547 len 4096 11/30 Write block 64 len 28672 data: 2 2 2 2 2 2 2 12/30 Read block 580 len 32768 13/30 Read block 276 len 40960 14/30 Write block 327 len 32768 data: 4 4 4 4 2 2 2 2 15/30 Write block 338 len 24576 data: 2 2 2 2 2 2 16/30 Write block 54 len 12288 data: 3 2 2 17/30 Write block 414 len 12288 data: 2 2 2 18/30 Write block 367 len 40960 data: 2 2 2 2 2 2 2 2 2 2 19/30 Write block 351 len 20480 data: 2 2 2 2 2 20/30 Read block 64 len 4096 21/30 Read block 583 len 8192 22/30 Write block 158 len 8192 data: 2 2 23/30 Read block 547 len 20480 24/30 Read block 66 len 32768 25/30 Read block 579 len 28672 26/30 Read block 290 len 28672 27/30 Read block 378 len 16384 28/30 Write block 106 len 12288 data: 2 2 3 29/30 Write block 364 len 4096 data: 2 30/30 Read block 226 len 40960 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Sent Done false 1001 Acked Write 9 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) None Some(JobId(1000)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:49.512202837Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1218} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 T F T <--- 2 040-059 3 3 3 6 6 6 T F T <--- 3 060-079 1 1 1 1 1 1 T F T <--- 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 2 2 2 T F T <--- 6 120-139 2 2 2 2 2 2 T F T <--- 7 140-159 2 2 2 2 2 2 T F T <--- 8 160-179 2 2 2 4 4 4 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 2 2 2 5 5 5 F F F 14 280-299 2 2 2 3 3 3 T F T <--- 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 3 3 3 T F T <--- 17 340-359 1 1 1 1 1 1 T F T <--- 18 360-379 1 1 1 1 1 1 T F T <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 T F T <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 6 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 4 4 7 7 7 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 4 4 4 7 7 7 F F F 7 140-159 4 4 4 7 7 7 F F F 8 160-179 2 2 2 4 4 4 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 2 2 2 5 5 5 F F F 14 280-299 4 4 4 7 7 7 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 4 4 4 7 7 7 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 4 4 4 7 7 7 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 4, Max flush: 7 Begin loop 3 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.828398008Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840091991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840409525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840449423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840461969Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840480007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"Crucible c001466c-2051-426f-8e56-3ca086a98f6f has session id: d13b2d6c-6b4c-472e-9afb-45831145680d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840813799Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Upstairs opts: Upstairs UUID: c001466c-2051-426f-8e56-3ca086a98f6f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840840201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Crucible stats registered with UUID: c001466c-2051-426f-8e56-3ca086a98f6f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.840854718Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:53.840868234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.84105666Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841091335Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841107493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841218303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841234421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"c001466c-2051-426f-8e56-3ca086a98f6f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841249808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841262934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.8412768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841290976Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841304683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841318869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841333025Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"ds_connection connected from Ok(127.0.0.1:57453)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841366601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"ds_connection connected from Ok(127.0.0.1:51555)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841382708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"ds_connection connected from Ok(127.0.0.1:56918)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.841397785Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842077364Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842113451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842130378Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842492993Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842539244Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[0]R flush_numbers[0..12]: [1, 7, 7, 7, 1, 7, 7, 7, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.84259622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[0]R generation[0..12]: [1, 4, 4, 4, 1, 4, 4, 4, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842612218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842628625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[1]R flush_numbers[0..12]: [1, 7, 7, 7, 1, 7, 7, 7, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842643662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[1]R generation[0..12]: [1, 4, 4, 4, 1, 4, 4, 4, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842658549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842674986Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[2]R flush_numbers[0..12]: [1, 7, 7, 7, 1, 7, 7, 7, 4, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842689593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[2]R generation[0..12]: [1, 4, 4, 4, 1, 4, 4, 4, 2, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.8427045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842721067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Max found gen is 5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842734353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Generation requested: 6 >= found:5","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:00:53.84274846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Next flush: 8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842765928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842779424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"":"downstairs","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.84279392Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842806836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"0","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842822754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"1","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.84283666Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"client":"2","":"downstairs","session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842850756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842866874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"c001466c-2051-426f-8e56-3ca086a98f6f is now active with session: d13b2d6c-6b4c-472e-9afb-45831145680d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.84288203Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842894796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226,"session_id":"d13b2d6c-6b4c-472e-9afb-45831145680d"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842908372Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} {"msg":"Activated sub_volume c001466c-2051-426f-8e56-3ca086a98f6f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:53.842919978Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 503 len 20480 data: 2 2 2 2 2 02/30 Read block 113 len 12288 03/30 Read block 438 len 12288 04/30 Read block 211 len 36864 05/30 Read block 487 len 32768 06/30 Read block 338 len 28672 07/30 Read block 461 len 32768 08/30 Read block 285 len 28672 09/30 Flush 10/30 Write block 189 len 4096 data: 2 11/30 Write block 548 len 24576 data: 2 2 2 2 2 2 12/30 Read block 87 len 20480 13/30 Write block 279 len 16384 data: 2 2 2 3 14/30 Read block 569 len 36864 15/30 Read block 371 len 24576 16/30 Read block 339 len 36864 17/30 Write block 122 len 12288 data: 3 2 2 18/30 Write block 234 len 20480 data: 2 2 2 2 2 19/30 Write block 154 len 24576 data: 2 2 2 2 3 3 20/30 Read block 362 len 28672 21/30 Read block 6 len 12288 22/30 Write block 509 len 40960 data: 2 2 2 2 2 2 2 2 2 2 23/30 Write block 49 len 28672 data: 2 2 3 3 3 4 3 24/30 Read block 305 len 40960 25/30 Write block 89 len 12288 data: 2 2 2 26/30 Flush 27/30 Write block 277 len 4096 data: 2 28/30 Write block 374 len 16384 data: 3 3 3 2 29/30 Write block 574 len 36864 data: 2 2 2 2 2 2 2 2 2 30/30 Write block 66 len 28672 data: 3 3 3 3 3 2 2 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 28 2 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:53.854235488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1226} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 4 4 7 7 7 F F F 2 040-059 6 4 6 9 7 9 F F F <--- 3 060-079 4 4 4 7 7 7 T F T <--- 4 080-099 6 1 6 9 1 9 F F F <--- 5 100-119 4 4 4 7 7 7 F F F 6 120-139 6 4 6 9 7 9 F F F <--- 7 140-159 6 4 6 9 7 9 F F F <--- 8 160-179 2 2 2 4 4 4 F F F 9 180-199 6 2 6 9 3 9 F F F <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 6 1 6 9 1 9 F F F <--- 12 240-259 2 2 2 5 5 5 F F F 13 260-279 6 2 6 9 5 9 T F T <--- 14 280-299 6 4 6 9 7 9 F F F <--- 15 300-319 1 1 1 1 1 1 F F F 16 320-339 4 4 4 7 7 7 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 4 4 4 7 7 7 T F T <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 1 6 9 1 9 F T F <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 6 1 6 9 1 9 F F F <--- 28 560-579 1 1 1 1 1 1 T F T <--- 29 580-599 1 1 1 1 1 1 T F T <--- Max gen: 6, Max flush: 9 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 4 4 7 7 7 F F F 2 040-059 6 6 6 9 9 9 F F F 3 060-079 7 7 7 10 10 10 F F F 4 080-099 6 6 6 9 9 9 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 6 6 6 9 9 9 F F F 7 140-159 6 6 6 9 9 9 F F F 8 160-179 2 2 2 4 4 4 F F F 9 180-199 6 6 6 9 9 9 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 6 6 6 9 9 9 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 7 7 7 10 10 10 F F F 14 280-299 6 6 6 9 9 9 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 4 4 4 7 7 7 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 7 7 7 10 10 10 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 6 6 9 9 9 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 6 6 6 9 9 9 F F F 28 560-579 7 7 7 10 10 10 F F F 29 580-599 7 7 7 10 10 10 F F F Max gen: 7, Max flush: 10 Begin loop 4 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.177696818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.189492437Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.189821507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.189861675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.18987392Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.189891939Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"Crucible 244323c1-1faa-4271-abbb-edca44ebce29 has session id: 0e8bff8b-bf96-44e9-81c8-2788b11715ee","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190218267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Upstairs opts: Upstairs UUID: 244323c1-1faa-4271-abbb-edca44ebce29, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190244579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Crucible stats registered with UUID: 244323c1-1faa-4271-abbb-edca44ebce29","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190262247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:58.190275613Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190430154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190453534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190468051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190603142Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.19061944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"244323c1-1faa-4271-abbb-edca44ebce29 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190635897Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190648403Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190662579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190676286Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190689842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190703908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190718065Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:53413)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190777702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:37597)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190808876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:55509)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.190825143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.191513897Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.191551254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.191624497Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.191923443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.19193753Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[0]R flush_numbers[0..12]: [1, 7, 9, 10, 9, 7, 9, 9, 4, 9, 1, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192034314Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[0]R generation[0..12]: [1, 4, 6, 7, 6, 4, 6, 6, 2, 6, 1, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192050571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192067029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[1]R flush_numbers[0..12]: [1, 7, 9, 10, 9, 7, 9, 9, 4, 9, 1, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192082045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[1]R generation[0..12]: [1, 4, 6, 7, 6, 4, 6, 6, 2, 6, 1, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192096942Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.19211314Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[2]R flush_numbers[0..12]: [1, 7, 9, 10, 9, 7, 9, 9, 4, 9, 1, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192128026Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[2]R generation[0..12]: [1, 4, 6, 7, 6, 4, 6, 6, 2, 6, 1, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192146675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192162992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192176198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192196307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Next flush: 11","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192209614Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.19222317Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"":"downstairs","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192236936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192249462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"0","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192263518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"1","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"2026-03-14T05:00:58.192277705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"client":"2","":"downstairs","session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192295192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192308549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"244323c1-1faa-4271-abbb-edca44ebce29 is now active with session: 0e8bff8b-bf96-44e9-81c8-2788b11715ee","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192323465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192336371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234,"session_id":"0e8bff8b-bf96-44e9-81c8-2788b11715ee"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192349547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} {"msg":"Activated sub_volume 244323c1-1faa-4271-abbb-edca44ebce29","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:00:58.192360822Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 194 len 20480 data: 2 2 2 2 2 03/30 Write block 58 len 32768 data: 2 2 2 2 2 2 3 3 04/30 Write block 24 len 20480 data: 2 2 2 2 2 05/30 Write block 316 len 28672 data: 2 2 2 2 2 2 2 06/30 Write block 338 len 16384 data: 3 3 3 3 07/30 Flush 08/30 Write block 432 len 20480 data: 2 2 2 2 2 09/30 Write block 319 len 16384 data: 3 3 3 3 10/30 Write block 577 len 40960 data: 3 3 3 3 3 3 2 2 2 2 11/30 Write block 163 len 28672 data: 2 2 2 2 2 2 2 12/30 Read block 299 len 36864 13/30 Read block 477 len 36864 14/30 Read block 380 len 8192 15/30 Read block 209 len 16384 16/30 Write block 424 len 32768 data: 2 2 2 2 2 2 2 2 17/30 Read block 23 len 12288 18/30 Read block 265 len 4096 19/30 Write block 127 len 20480 data: 2 2 2 2 2 20/30 Write block 24 len 28672 data: 3 3 3 3 3 2 2 21/30 Write block 223 len 36864 data: 2 2 2 2 2 2 2 2 2 22/30 Read block 435 len 4096 23/30 Read block 560 len 24576 24/30 Write block 49 len 12288 data: 3 3 4 25/30 Write block 66 len 12288 data: 4 4 4 26/30 Flush 27/30 Write block 35 len 32768 data: 3 3 2 2 2 2 2 2 28/30 Write block 493 len 40960 data: 2 2 2 2 2 2 2 2 2 2 29/30 Write block 579 len 12288 data: 4 4 4 30/30 Write block 90 len 24576 data: 3 3 2 2 2 2 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 26 32 Done 26 26 3 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:00:58.203310846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1234} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 8 8 4 13 13 7 T T T <--- 2 040-059 8 8 6 13 13 9 T T T <--- 3 060-079 8 8 7 13 13 10 F F T <--- 4 080-099 6 6 6 9 9 9 T T F <--- 5 100-119 4 4 4 7 7 7 F F F 6 120-139 8 8 6 13 13 9 F F F <--- 7 140-159 6 6 6 9 9 9 F F F 8 160-179 8 8 2 13 13 4 F F F <--- 9 180-199 8 8 6 12 12 9 F F T <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 8 8 6 13 13 9 F F F <--- 12 240-259 2 2 2 5 5 5 F F F 13 260-279 7 7 7 10 10 10 F F F 14 280-299 6 6 6 9 9 9 F F F 15 300-319 8 8 1 13 13 1 F F F <--- 16 320-339 8 8 4 13 13 7 F F F <--- 17 340-359 8 8 4 12 12 7 F F F <--- 18 360-379 7 7 7 10 10 10 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 8 8 1 13 13 1 F F F <--- 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 T T F <--- 25 500-519 6 6 6 9 9 9 T T F <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 6 6 6 9 9 9 F F F 28 560-579 8 8 7 13 13 10 T T F <--- 29 580-599 8 8 7 13 13 10 T T F <--- Max gen: 8, Max flush: 13 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 9 9 9 14 14 14 F F F 2 040-059 9 9 9 14 14 14 F F F 3 060-079 8 8 8 13 13 13 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 8 8 8 13 13 13 F F F 7 140-159 6 6 6 9 9 9 F F F 8 160-179 8 8 8 13 13 13 F F F 9 180-199 8 8 8 12 12 12 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 8 8 8 13 13 13 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 7 7 7 10 10 10 F F F 14 280-299 6 6 6 9 9 9 F F F 15 300-319 8 8 8 13 13 13 F F F 16 320-339 8 8 8 13 13 13 F F F 17 340-359 8 8 8 12 12 12 F F F 18 360-379 7 7 7 10 10 10 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 8 8 8 13 13 13 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 14 14 14 F F F 25 500-519 9 9 9 14 14 14 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 6 6 6 9 9 9 F F F 28 560-579 9 9 9 14 14 14 F F F 29 580-599 9 9 9 14 14 14 F F F Max gen: 9, Max flush: 14 Begin loop 5 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.53387751Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.545616643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.545965852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546004839Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546017655Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546035773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"Crucible bb5d49db-3e0b-4be4-8aab-8cd55734bdfe has session id: 16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546388604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Upstairs opts: Upstairs UUID: bb5d49db-3e0b-4be4-8aab-8cd55734bdfe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54642347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Crucible stats registered with UUID: bb5d49db-3e0b-4be4-8aab-8cd55734bdfe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546439597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:02.546452853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54659852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546634876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546650643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546756762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546772119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"bb5d49db-3e0b-4be4-8aab-8cd55734bdfe active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546793428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546806254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546820241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546839929Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"2","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546853876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546867782Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546881949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"ds_connection connected from Ok(127.0.0.1:65531)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54692967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"ds_connection connected from Ok(127.0.0.1:54785)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.546947238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"ds_connection connected from Ok(127.0.0.1:52870)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.547084721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.547664455Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.547707624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.547757437Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"2","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54804935Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54809255Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[0]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 13, 9, 13, 12, 1, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548156028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[0]R generation[0..12]: [1, 9, 9, 8, 9, 4, 8, 6, 8, 8, 1, 8]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548172786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548189304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[1]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 13, 9, 13, 12, 1, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548204991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[1]R generation[0..12]: [1, 9, 9, 8, 9, 4, 8, 6, 8, 8, 1, 8]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548220328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548237616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[2]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 13, 9, 13, 12, 1, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548252843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[2]R generation[0..12]: [1, 9, 9, 8, 9, 4, 8, 6, 8, 8, 1, 8]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548267379Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548283667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548297013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P",Wait for a query_work_queue command to finish before sending IO "pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548317262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Next flush: 15","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548330308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548343564Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"":"downstairs","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54835775Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548370386Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"0","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548386573Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"1","":"downstairs","session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.54840074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548417578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548431164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"bb5d49db-3e0b-4be4-8aab-8cd55734bdfe is now active with session: 16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548446111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548458866Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242,"session_id":"16ee4e63-87f6-46c8-8e4e-66a9b7f2abe8"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548472072Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} {"msg":"Activated sub_volume bb5d49db-3e0b-4be4-8aab-8cd55734bdfe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:02.548483498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 516 len 4096 data: 3 02/30 Write block 178 len 36864 data: 3 2 3 3 3 3 3 3 2 03/30 Read block 102 len 12288 04/30 Read block 240 len 16384 05/30 Read block 341 len 4096 06/30 Write block 260 len 28672 data: 3 3 2 2 2 2 3 07/30 Write block 343 len 36864 data: 3 2 2 2 2 2 2 2 3 08/30 Write block 531 len 12288 data: 2 2 2 09/30 Write block 121 len 20480 data: 3 4 3 3 2 10/30 Read block 51 len 28672 11/30 Flush 12/30 Write block 206 len 28672 data: 2 2 2 2 2 2 2 13/30 Write block 284 len 40960 data: 3 3 4 3 3 3 2 2 2 2 14/30 Write block 135 len 24576 data: 3 3 3 3 3 3 15/30 Write block 267 len 12288 data: 3 3 3 16/30 Read block 528 len 8192 17/30 Write block 216 len 40960 data: 2 2 2 2 2 2 2 3 3 3 18/30 Read block 350 len 32768 19/30 Write block 464 len 36864 data: 2 2 2 2 2 2 2 2 2 20/30 Read block 449 len 36864 21/30 Read block 535 len 40960 22/30 Write block 434 len 4096 data: 3 23/30 Read block 20 len 28672 24/30 Read block 543 len 32768 25/30 Read block 61 len 4096 26/30 Write block 315 len 20480 data: 2 3 3 3 4 27/30 Write block 422 len 24576 data: 2 2 3 3 3 3 28/30 Write block 559 len 16384 data: 2 2 2 2 29/30 Read block 551 len 24576 30/30 Read block 159 len 12288 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Write 9 Done Done Sent false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:02.560124036Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1242} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 9 9 9 14 14 14 F F F 2 040-059 9 9 9 14 14 14 F F F 3 060-079 8 8 8 13 13 13 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 10 10 8 15 15 13 T T F <--- 7 140-159 6 6 6 9 9 9 T T F <--- 8 160-179 10 10 8 15 15 13 F F F <--- 9 180-199 10 10 8 15 15 12 F F F <--- 10 200-219 1 1 1 1 1 1 T T F <--- 11 220-239 8 8 8 13 13 13 T T F <--- 12 240-259 2 2 2 5 5 5 F F F 13 260-279 10 10 7 15 15 10 T T F <--- 14 280-299 6 6 6 9 9 9 T T F <--- 15 300-319 8 8 8 13 13 13 T T F <--- 16 320-339 8 8 8 13 13 13 F F F 17 340-359 10 10 8 15 15 12 F F F <--- 18 360-379 7 7 7 10 10 10 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 8 8 8 13 13 13 T T F <--- 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 T T F <--- 24 480-499 9 9 9 14 14 14 F F F 25 500-519 10 10 9 15 15 14 F F F <--- 26 520-539 10 10 1 15 15 1 F F F <--- 27 540-559 6 6 6 9 9 9 T T F <--- 28 560-579 9 9 9 14 14 14 T T F <--- 29 580-599 9 9 9 14 14 14 F F F Max gen: 10, Max flush: 15 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 9 9 9 14 14 14 F F F 2 040-059 9 9 9 14 14 14 F F F 3 060-079 8 8 8 13 13 13 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 10 10 10 15 15 15 F F F 9 180-199 10 10 10 15 15 15 F F F 10 200-219 11 11 11 16 16 16 F F F 11 220-239 11 11 11 16 16 16 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 11 11 11 16 16 16 F F F 14 280-299 11 11 11 16 16 16 F F F 15 300-319 11 11 11 16 16 16 F F F 16 320-339 8 8 8 13 13 13 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 7 7 7 10 10 10 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 11 11 11 16 16 16 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 11 11 11 16 16 16 F F F 24 480-499 9 9 9 14 14 14 F F F 25 500-519 10 10 10 15 15 15 F F F 26 520-539 10 10 10 15 15 15 F F F 27 540-559 11 11 11 16 16 16 F F F 28 560-579 11 11 11 16 16 16 F F F 29 580-599 9 9 9 14 14 14 F F F Max gen: 11, Max flush: 16 Begin loop 6 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.891449304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903101767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903426475Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903469175Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903482211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903500739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"Crucible 7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92 has session id: e7b397d7-84e6-44f5-9bfd-e5821bb0883f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903840554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Upstairs opts: Upstairs UUID: 7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903877791Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Crucible stats registered with UUID: 7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.903894048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:06.903907584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904054961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904079282Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904094299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904223048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904238445Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904253262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904265747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904279784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.90429373Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904307306Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904321853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904336139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"ds_connection connected from Ok(127.0.0.1:39024)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904429522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"ds_connection connected from Ok(127.0.0.1:61191)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904460086Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43696)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.904526106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905183495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905203164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905219171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905555244Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905599234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[0]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 16, 16, 15, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.9056558Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[0]R generation[0..12]: [1, 9, 9, 8, 9, 4, 11, 11, 10, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905672578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905688945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[1]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 16, 16, 15, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905704432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[1]R generation[0..12]: [1, 9, 9, 8, 9, 4, 11, 11, 10, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905719359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905736096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[2]R flush_numbers[0..12]: [1, 14, 14, 13, 14, 7, 16, 16, 15, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905751213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[2]R generation[0..12]: [1, 9, 9, 8, 9, 4, 11, 11, 10, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905768331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905784739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905798305Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905817814Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Next flush: 17","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.90583107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905844866Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"":"downstairs","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905858972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905871738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"0","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905886155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"1","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905900191Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"client":"2","":"downstairs","session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"All required reconciliation work is completed","v":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905917259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905931025Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92 is now active with session: e7b397d7-84e6-44f5-9bfd-e5821bb0883f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905946152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905959048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250,"session_id":"e7b397d7-84e6-44f5-9bfd-e5821bb0883f"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905972784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} {"msg":"Activated sub_volume 7f6ee7a2-9f3a-4ab3-9efc-1778f5dc5d92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:06.905984049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 456 len 12288 data: 2 2 2 02/30 Write block 167 len 8192 data: 3 3 03/30 Read block 451 len 28672 04/30 Write block 553 len 28672 data: 3 2 2 2 2 2 3 05/30 Write block 285 len 4096 data: 4 06/30 Read block 229 len 12288 07/30 Read block 423 len 28672 08/30 Write block 273 len 28672 data: 3 2 2 2 3 2 3 09/30 Write block 31 len 40960 data: 2 2 3 3 4 4 3 3 3 3 10/30 Read block 265 len 4096 11/30 Write block 594 len 24576 data: 2 2 2 2 2 2 12/30 Write block 369 len 8192 data: 3 3 13/30 Write block 75 len 4096 data: 2 14/30 Write block 477 len 36864 data: 2 2 2 2 2 2 2 2 2 15/30 Read block 6 len 12288 16/30 Read block 233 len 16384 17/30 Write block 326 len 4096 data: 4 18/30 Read block 315 len 24576 19/30 Write block 412 len 16384 data: 2 2 3 3 20/30 Read block 579 len 36864 21/30 Read block 223 len 8192 22/30 Read block 474 len 8192 23/30 Read block 379 len 8192 24/30 Read block 94 len 12288 25/30 Read block 243 len 8192 26/30 Read block 466 len 32768 27/30 Write block 351 len 32768 data: 4 3 3 3 3 2 2 2 28/30 Read block 378 len 32768 29/30 Write block 552 len 20480 data: 3 4 3 3 3 30/30 Write block 448 len 24576 data: 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:06.917143548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1250} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 9 9 9 14 14 14 T F T <--- 2 040-059 9 9 9 14 14 14 T F T <--- 3 060-079 8 8 8 13 13 13 T F T <--- 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 10 10 10 15 15 15 T F T <--- 9 180-199 10 10 10 15 15 15 F F F 10 200-219 11 11 11 16 16 16 F F F 11 220-239 11 11 11 16 16 16 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 11 11 11 16 16 16 T F T <--- 14 280-299 11 11 11 16 16 16 T F T <--- 15 300-319 11 11 11 16 16 16 F F F 16 320-339 8 8 8 13 13 13 T F T <--- 17 340-359 10 10 10 15 15 15 T F T <--- 18 360-379 7 7 7 10 10 10 T F T <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 4 4 4 7 7 7 T F T <--- 21 420-439 11 11 11 16 16 16 F F F 22 440-459 1 1 1 1 1 1 T F T <--- 23 460-479 11 11 11 16 16 16 T F T <--- 24 480-499 9 9 9 14 14 14 T F T <--- 25 500-519 10 10 10 15 15 15 F F F 26 520-539 10 10 10 15 15 15 F F F 27 540-559 11 11 11 16 16 16 T F T <--- 28 560-579 11 11 11 16 16 16 F F F 29 580-599 9 9 9 14 14 14 T F T <--- Max gen: 11, Max flush: 16 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 12 12 12 17 17 17 F F F 3 060-079 12 12 12 17 17 17 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 12 12 12 17 17 17 F F F 9 180-199 10 10 10 15 15 15 F F F 10 200-219 11 11 11 16 16 16 F F F 11 220-239 11 11 11 16 16 16 F F F 12 240-259 2 2 2 5 5 5 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 12 12 12 17 17 17 F F F 15 300-319 11 11 11 16 16 16 F F F 16 320-339 12 12 12 17 17 17 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 12 12 12 17 17 17 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 12 12 12 17 17 17 F F F 21 420-439 11 11 11 16 16 16 F F F 22 440-459 12 12 12 17 17 17 F F F 23 460-479 12 12 12 17 17 17 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 10 10 10 15 15 15 F F F 26 520-539 10 10 10 15 15 15 F F F 27 540-559 12 12 12 17 17 17 F F F 28 560-579 11 11 11 16 16 16 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 12, Max flush: 17 Begin loop 7 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.252675472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:11.263346649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.263404675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.273802217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274101383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274140131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274152707Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274170745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"Crucible 2e785627-3e53-4201-b005-323b18f87c59 has session id: d54af272-ecff-47e8-82bd-7e36562fa92c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274527537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Upstairs opts: Upstairs UUID: 2e785627-3e53-4201-b005-323b18f87c59, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274562163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Crucible stats registered with UUID: 2e785627-3e53-4201-b005-323b18f87c59","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274578281Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:11.274591406Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274728139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274757942Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274772609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274904189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274918815Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"2e785627-3e53-4201-b005-323b18f87c59 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274934543Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274947318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274961545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274975541Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.274989378Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275004234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275018831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"ds_connection connected from Ok(127.0.0.1:48891)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275033898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"ds_connection connected from Ok(127.0.0.1:47020)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275048955Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45912)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275152622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275828029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275858393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.275947974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276189144Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276205971Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[0]R flush_numbers[0..12]: [1, 17, 17, 17, 14, 7, 16, 16, 17, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276342303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[0]R generation[0..12]: [1, 12, 12, 12, 9, 4, 11, 11, 12, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276359891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276376289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[1]R flush_numbers[0..12]: [1, 17, 17, 17, 14, 7, 16, 16, 17, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276391556Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[1]R generation[0..12]: [1, 12, 12, 12, 9, 4, 11, 11, 12, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276406813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.27642292Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[2]R flush_numbers[0..12]: [1, 17, 17, 17, 14, 7, 16, 16, 17, 15, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276438627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[2]R generation[0..12]: [1, 12, 12, 12, 9, 4, 11, 11, 12, 10, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276454064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276470522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Max found gen is 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276484068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":Wait for a query_work_queue command to finish before sending IO "downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Generation requested: 14 >= found:13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276504867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276517913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276531519Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"":"downstairs","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276548347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276561213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"0","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276575269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"1","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:01:11.276588936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"client":"2","":"downstairs","session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276605853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276619069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"2e785627-3e53-4201-b005-323b18f87c59 is now active with session: d54af272-ecff-47e8-82bd-7e36562fa92c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276633916Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276646662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258,"session_id":"d54af272-ecff-47e8-82bd-7e36562fa92c"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276660068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} {"msg":"Activated sub_volume 2e785627-3e53-4201-b005-323b18f87c59","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:11.276671383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 537 len 40960 data: 2 2 2 2 2 2 2 2 2 2 02/30 Read block 195 len 16384 03/30 Write block 17 len 4096 data: 2 04/30 Read block 417 len 24576 05/30 Write block 179 len 20480 data: 3 4 4 4 4 06/30 Write block 586 len 32768 data: 3 2 2 2 2 2 2 2 07/30 Read block 374 len 28672 08/30 Write block 59 len 36864 data: 3 3 3 3 3 4 4 5 5 09/30 Write block 514 len 4096 data: 3 10/30 Flush 11/30 Write block 230 len 36864 data: 3 3 2 2 3 3 3 3 3 12/30 Read block 429 len 12288 13/30 Read block 263 len 40960 14/30 Read block 421 len 12288 15/30 Write block 42 len 12288 data: 3 2 2 16/30 Write block 77 len 4096 data: 2 17/30 Write block 252 len 20480 data: 2 2 3 3 3 18/30 Flush 19/30 Write block 228 len 20480 data: 3 3 4 4 3 20/30 Read block 158 len 24576 21/30 Write block 364 len 20480 data: 3 2 2 3 3 22/30 Read block 186 len 36864 23/30 Read block 522 len 24576 24/30 Read block 200 len 28672 25/30 Read block 5 len 40960 26/30 Write block 314 len 40960 data: 2 3 4 4 4 5 4 4 4 3 27/30 Read block 420 len 20480 28/30 Write block 53 len 24576 data: 4 5 4 3 2 3 29/30 Read block 503 len 24576 30/30 Read block 261 len 40960 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:11.289982612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1258} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 14 14 1 18 18 F F F <--- 1 020-039 12 12 12 17 17 17 F F F 2 040-059 12 14 14 17 19 19 F T T <--- 3 060-079 12 14 14 17 19 19 F F F <--- 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 12 14 14 17 18 18 F F F <--- 9 180-199 10 14 14 15 18 18 F F F <--- 10 200-219 11 11 11 16 16 16 F F F 11 220-239 11 14 14 16 19 19 F T T <--- 12 240-259 2 14 14 5 19 19 F F F <--- 13 260-279 12 12 12 17 17 17 F F F 14 280-299 12 12 12 17 17 17 F F F 15 300-319 11 11 11 16 16 16 F T T <--- 16 320-339 12 12 12 17 17 17 F T T <--- 17 340-359 12 12 12 17 17 17 F F F 18 360-379 12 12 12 17 17 17 F T T <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 12 12 12 17 17 17 F F F 21 420-439 11 11 11 16 16 16 F F F 22 440-459 12 12 12 17 17 17 F F F 23 460-479 12 12 12 17 17 17 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 10 14 14 15 18 18 F F F <--- 26 520-539 10 14 14 15 18 18 T F F <--- 27 540-559 12 14 14 17 18 18 T F F <--- 28 560-579 11 11 11 16 16 16 F F F 29 580-599 12 14 14 17 18 18 F F F <--- Max gen: 14, Max flush: 19 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 18 18 18 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 14 14 14 18 18 18 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 11 11 11 16 16 16 F F F 11 220-239 15 15 15 20 20 20 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 12 12 12 17 17 17 F F F 15 300-319 15 15 15 20 20 20 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 15 15 15 20 20 20 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 12 12 12 17 17 17 F F F 21 420-439 11 11 11 16 16 16 F F F 22 440-459 12 12 12 17 17 17 F F F 23 460-479 12 12 12 17 17 17 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 14 14 14 18 18 18 F F F 26 520-539 14 14 14 18 18 18 F F F 27 540-559 14 14 14 18 18 18 F F F 28 560-579 11 11 11 16 16 16 F F F 29 580-599 14 14 14 18 18 18 F F F Max gen: 15, Max flush: 20 Begin loop 8 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.625529892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.637496349Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.637804219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.637842126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.637854922Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.6378732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"Crucible c03b5e3e-9938-41b8-a763-8136a489d083 has session id: 32a439d6-b2bb-408f-ab49-778fb03ee49c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638210324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Upstairs opts: Upstairs UUID: c03b5e3e-9938-41b8-a763-8136a489d083, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63824584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Crucible stats registered with UUID: c03b5e3e-9938-41b8-a763-8136a489d083","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638263208Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:15.638277024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638435926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638469872Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638485039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638586625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638601792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"c03b5e3e-9938-41b8-a763-8136a489d083 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638616829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638629674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638643581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638657517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638671173Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63868539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638702247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"ds_connection connected from Ok(127.0.0.1:35828)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638722136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"ds_connection connected from Ok(127.0.0.1:52356)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.638736783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"ds_connection connected from Ok(127.0.0.1:33031)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63875114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.639475249Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.639512546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.639529884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63985055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63989338Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[0]R flush_numbers[0..12]: [18, 17, 20, 19, 14, 7, 16, 16, 18, 18, 16, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.639966593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[0]R generation[0..12]: [14, 12, 15, 14, 9, 4, 11, 11, 14, 14, 11, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.63998326Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.639999528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[1]R flush_numbers[0..12]: [18, 17, 20, 19, 14, 7, 16, 16, 18, 18, 16, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640015185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[1]R generation[0..12]: [14, 12, 15, 14, 9, 4, 11, 11, 14, 14, 11, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640030572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640046879Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[2]R flush_numbers[0..12]: [18, 17, 20, 19, 14, 7, 16, 16, 18, 18, 16, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640065598Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[2]R generation[0..12]: [14, 12, 15, 14, 9, 4, 11, 11, 14, 14, 11, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640080945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640097092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640110498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640130027Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Next flush: 21","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640143413Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640156709Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"":"downstairs","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640170696Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640183141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"0","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640196778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"1","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640210574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"client":"2","":"downstairs","session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.64022481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640240367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"c03b5e3e-9938-41b8-a763-8136a489d083 is now active with session: 32a439d6-b2bb-408f-ab49-778fb03ee49c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640255154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.64026786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266,"session_id":"32a439d6-b2bb-408f-ab49-778fb03ee49c"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640282927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} {"msg":"Activated sub_volume c03b5e3e-9938-41b8-a763-8136a489d083","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:15.640294702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 210 len 8192 data: 3 3 02/30 Write block 532 len 20480 data: 3 3 2 2 2 03/30 Write block 394 len 32768 data: 2 2 2 2 2 2 2 2 04/30 Write block 569 len 16384 data: 2 2 2 2 05/30 Read block 272 len 24576 06/30 Read block 109 len 4096 07/30 Read block 57 len 32768 08/30 Read block 41 len 40960 09/30 Write block 550 len 36864 data: 3 3 4 5 4 4 4 3 3 10/30 Write block 479 len 20480 data: 3 3 3 3 3 11/30 Write block 493 len 8192 data: 3 3 12/30 Read block 198 len 8192 13/30 Read block 122 len 12288 14/30 Write block 289 len 4096 data: 4 15/30 Read block 94 len 4096 16/30 Write block 438 len 20480 data: 2 2 2 2 2 17/30 Read block 479 len 16384 18/30 Read block 456 len 24576 19/30 Read block 123 len 32768 20/30 Read block 504 len 36864 21/30 Read block 533 len 36864 22/30 Write block 177 len 12288 data: 3 4 4 23/30 Read block 561 len 8192 24/30 Read block 81 len 4096 25/30 Flush 26/30 Write block 306 len 8192 data: 2 2 27/30 Read block 97 len 4096 28/30 Write block 86 len 4096 data: 2 29/30 Read block 451 len 12288 30/30 Read block 548 len 40960 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 4 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Read 10 Done Done Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 22 1 23 Done 30 8 29 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:15.650760065Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1266} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 18 18 18 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 9 9 9 14 14 14 T F T <--- 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 16 14 16 21 18 21 F F F <--- 9 180-199 14 14 14 18 18 18 F F F 10 200-219 16 11 16 21 16 21 F T F <--- 11 220-239 15 15 15 20 20 20 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 16 12 16 21 17 21 F F F <--- 15 300-319 15 15 15 20 20 20 T F T <--- 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 15 15 15 20 20 20 F F F 19 380-399 16 1 16 21 1 21 F T F <--- 20 400-419 16 12 16 21 17 21 F T F <--- 21 420-439 16 11 16 21 16 21 F F F <--- 22 440-459 16 12 16 21 17 21 F F F <--- 23 460-479 16 12 16 21 17 21 F F F <--- 24 480-499 16 12 16 21 17 21 F F F <--- 25 500-519 14 14 14 18 18 18 F F F 26 520-539 16 14 16 21 18 21 F T F <--- 27 540-559 16 14 16 21 18 21 F F F <--- 28 560-579 16 11 16 21 16 21 F T F <--- 29 580-599 14 14 14 18 18 18 F F F Max gen: 16, Max flush: 21 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 18 18 18 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 17 17 17 22 22 22 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 F F F 7 140-159 11 11 11 16 16 16 F F F 8 160-179 16 16 16 21 21 21 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 16 16 16 21 21 21 F F F 11 220-239 15 15 15 20 20 20 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 16 16 16 21 21 21 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 15 15 15 20 20 20 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 16 16 16 21 21 21 F F F 21 420-439 16 16 16 21 21 21 F F F 22 440-459 16 16 16 21 21 21 F F F 23 460-479 16 16 16 21 21 21 F F F 24 480-499 16 16 16 21 21 21 F F F 25 500-519 14 14 14 18 18 18 F F F 26 520-539 16 16 16 21 21 21 F F F 27 540-559 16 16 16 21 21 21 F F F 28 560-579 16 16 16 21 21 21 F F F 29 580-599 14 14 14 18 18 18 F F F Max gen: 17, Max flush: 22 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.975690253Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.987378723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.987691526Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.987733365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.987746451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.987764509Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"Crucible a5d58f12-10d5-4742-b169-8157cd734237 has session id: f2cb5ab2-7c0d-4676-8a89-8a0352627591","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.9880955Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Upstairs opts: Upstairs UUID: a5d58f12-10d5-4742-b169-8157cd734237, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988130405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Crucible stats registered with UUID: a5d58f12-10d5-4742-b169-8157cd734237","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988146203Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:19.988159269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988318331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988352486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988368064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988450031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988467399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"a5d58f12-10d5-4742-b169-8157cd734237 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988490399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988503505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988517522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988531478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988544934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988559061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988573207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"ds_connection connected from Ok(127.0.0.1:46642)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988606812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"ds_connection connected from Ok(127.0.0.1:53965)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988622659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"ds_connection connected from Ok(127.0.0.1:41430)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.988746926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989343928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989380594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989404185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989704632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.98972268Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[0]R flush_numbers[0..12]: [18, 17, 20, 19, 22, 7, 16, 16, 21, 18, 21, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989828478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[0]R generation[0..12]: [14, 12, 15, 14, 17, 4, 11, 11, 16, 14, 16, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989845566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989862234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[1]R flush_numbers[0..12]: [18, 17, 20, 19, 22, 7, 16, 16, 21, 18, 21, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989877541Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[1]R generation[0..12]: [14, 12, 15, 14, 17, 4, 11, 11, 16, 14, 16, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989892748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989908965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[2]R flush_numbers[0..12]: [18, 17, 20, 19, 22, 7, 16, 16, 21, 18, 21, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989924412Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[2]R generation[0..12]: [14, 12, 15, 14, 17, 4, 11, 11, 16, 14, 16, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989939669Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989955826Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989969262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.989988561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Next flush: 23","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990002217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990015754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"":"downstairs","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990032111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990044927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"0","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990059483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"1","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.99007354Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"client":"2","":"downstairs","session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990087606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Set Downstairs and Upstairs active after reconciliation"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990103704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"a5d58f12-10d5-4742-b169-8157cd734237 is now active with session: f2cb5ab2-7c0d-4676-8a89-8a0352627591","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.99011865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990131506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274,"session_id":"f2cb5ab2-7c0d-4676-8a89-8a0352627591"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990144922Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} {"msg":"Activated sub_volume a5d58f12-10d5-4742-b169-8157cd734237","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:19.990156958Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 270 len 20480 data: 3 3 3 4 3 02/30 Read block 145 len 28672 03/30 Read block 330 len 32768 04/30 Write block 86 len 36864 data: 3 2 2 3 4 4 3 3 3 05/30 Read block 368 len 24576 06/30 Read block 81 len 4096 07/30 Read block 178 len 4096 08/30 Write block 438 len 28672 data: 3 3 3 3 3 2 2 09/30 Write block 134 len 20480 data: 3 4 4 4 4 10/30 Read block 226 len 32768 11/30 Write block 478 len 16384 data: 3 4 4 4 12/30 Read block 355 len 28672 13/30 Write block 286 len 36864 data: 5 4 4 5 3 3 3 3 2 14/30 Write block 541 len 8192 data: 3 3 15/30 Write block 482 len 40960 data: 4 4 3 3 2 2 2 2 2 2 16/30 Write block 543 len 24576 data: 3 3 3 3 2 3 17/30 Read block 500 len 8192 18/30 Write block 211 len 8192 data: 4 3 19/30 Read block 267 len 24576 20/30 Read block 167 len 12288 21/30 Write block 147 len 20480 data: 2 2 2 2 2 22/30 Read block 114 len 32768 23/30 Read block 428 len 36864 24/30 Write block 594 len 16384 data: 3 3 3 3 25/30 Read block 398 len 36864 26/30 Write block 8 len 36864 data: 2 2 2 2 2 2 2 2 2 27/30 Read block 352 len 32768 28/30 Read block 454 len 36864 29/30 Write block 470 len 12288 data: 3 3 3 30/30 Read block 2 len 40960 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 0 25 Done 29 6 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:20.00297041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1274} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 18 18 18 T F T <--- 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 17 17 17 22 22 22 T T T 5 100-119 4 4 4 7 7 7 F F F 6 120-139 11 11 11 16 16 16 T F T <--- 7 140-159 11 11 11 16 16 16 T F T <--- 8 160-179 16 16 16 21 21 21 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 16 16 16 21 21 21 T F T <--- 11 220-239 15 15 15 20 20 20 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 T T T 14 280-299 16 16 16 21 21 21 T F T <--- 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 15 15 15 20 20 20 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 16 16 16 21 21 21 F F F 21 420-439 16 16 16 21 21 21 T F T <--- 22 440-459 16 16 16 21 21 21 T F T <--- 23 460-479 16 16 16 21 21 21 T F T <--- 24 480-499 16 16 16 21 21 21 T F T <--- 25 500-519 14 14 14 18 18 18 F F F 26 520-539 16 16 16 21 21 21 F F F 27 540-559 16 16 16 21 21 21 T F T <--- 28 560-579 16 16 16 21 21 21 F F F 29 580-599 14 14 14 18 18 18 T F T <--- Max gen: 17, Max flush: 22 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 23 23 23 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 18 18 18 23 23 23 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 18 18 18 23 23 23 F F F 7 140-159 18 18 18 23 23 23 F F F 8 160-179 16 16 16 21 21 21 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 15 15 15 20 20 20 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 F F F 18 360-379 15 15 15 20 20 20 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 16 16 16 21 21 21 F F F 21 420-439 18 18 18 23 23 23 F F F 22 440-459 18 18 18 23 23 23 F F F 23 460-479 18 18 18 23 23 23 F F F 24 480-499 18 18 18 23 23 23 F F F 25 500-519 14 14 14 18 18 18 F F F 26 520-539 16 16 16 21 21 21 F F F 27 540-559 18 18 18 23 23 23 F F F 28 560-579 16 16 16 21 21 21 F F F 29 580-599 18 18 18 23 23 23 F F F Max gen: 18, Max flush: 23 Begin loop 10 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.330391662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342003757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342334097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342376296Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342389292Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.34240719Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"Crucible 1f628f9a-0e7a-4b29-9dca-b4900f86935f has session id: e3e5223b-d2a2-4c59-a312-0a37dd4d7395","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342741623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Upstairs opts: Upstairs UUID: 1f628f9a-0e7a-4b29-9dca-b4900f86935f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342777169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Crucible stats registered with UUID: 1f628f9a-0e7a-4b29-9dca-b4900f86935f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342792956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:24.342806342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.34295585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.342986194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343011796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343122766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343138043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"1f628f9a-0e7a-4b29-9dca-b4900f86935f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343156111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343169347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343183624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.34319734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343229535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343252575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343267002Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"ds_connection connected from Ok(127.0.0.1:45099)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343314243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"ds_connection connected from Ok(127.0.0.1:65130)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343465742Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"ds_connection connected from Ok(127.0.0.1:64074)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.343493325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344045596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344064475Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344080932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344421767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344467358Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[0]R flush_numbers[0..12]: [23, 17, 20, 19, 23, 7, 23, 23, 21, 18, 23, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344513149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[0]R generation[0..12]: [18, 12, 15, 14, 18, 4, 18, 18, 16, 14, 18, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344530477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344547264Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[1]R flush_numbers[0..12]: [23, 17, 20, 19, 23, 7, 23, 23, 21, 18, 23, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344562651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[1]R generation[0..12]: [18, 12, 15, 14, 18, 4, 18, 18, 16, 14, 18, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344578048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344600128Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[2]R flush_numbers[0..12]: [23, 17, 20, 19, 23, 7, 23, 23, 21, 18, 23, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344618817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[2]R generation[0..12]: [18, 12, 15, 14, 18, 4, 18, 18, 16, 14, 18, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344634494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344650731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Max found gen is 19","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344664338Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Generation requested: 20 >= found:19","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344678114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Next flush: 24","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2026-03-14T05:01:24.34469113Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344711209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"":"downstairs","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344725666Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344738181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"0","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344752728Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"1","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344766624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"client":"2","":"downstairs","session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344780911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:01:24.344794567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"1f628f9a-0e7a-4b29-9dca-b4900f86935f is now active with session: e3e5223b-d2a2-4c59-a312-0a37dd4d7395","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344811415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.34482422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282,"session_id":"e3e5223b-d2a2-4c59-a312-0a37dd4d7395"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344839547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} {"msg":"Activated sub_volume 1f628f9a-0e7a-4b29-9dca-b4900f86935f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:24.344851283Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 569 len 8192 data: 3 3 02/30 Write block 407 len 8192 data: 2 2 03/30 Write block 572 len 40960 data: 3 2 3 3 3 4 4 5 5 5 04/30 Read block 405 len 4096 05/30 Write block 225 len 12288 data: 4 3 3 06/30 Read block 345 len 12288 07/30 Write block 539 len 32768 data: 3 3 4 4 4 4 4 4 08/30 Read block 347 len 36864 09/30 Read block 524 len 16384 10/30 Read block 346 len 36864 11/30 Write block 565 len 20480 data: 2 2 2 2 4 12/30 Write block 590 len 12288 data: 3 3 3 13/30 Write block 543 len 36864 data: 5 5 5 5 3 4 3 4 4 14/30 Write block 510 len 24576 data: 3 3 3 3 4 3 15/30 Read block 15 len 28672 16/30 Write block 543 len 24576 data: 6 6 6 6 4 5 17/30 Write block 406 len 24576 data: 2 3 3 2 2 2 18/30 Read block 34 len 20480 19/30 Read block 456 len 12288 20/30 Write block 154 len 28672 data: 3 3 3 3 4 4 2 21/30 Write block 463 len 36864 data: 2 3 3 3 3 3 3 4 4 22/30 Read block 381 len 36864 23/30 Write block 498 len 32768 data: 3 3 3 3 3 3 3 3 24/30 Flush 25/30 Write block 535 len 20480 data: 3 3 3 3 4 26/30 Write block 108 len 24576 data: 4 2 3 3 3 3 27/30 Write block 6 len 4096 data: 2 28/30 Write block 9 len 4096 data: 3 29/30 Read block 15 len 4096 30/30 Write block 354 len 28672 data: 4 4 3 3 3 2 2 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 1 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:24.35668865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1282} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 23 23 23 T F T <--- 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 18 18 18 23 23 23 F F F 5 100-119 4 4 4 7 7 7 T F T <--- 6 120-139 18 18 18 23 23 23 F F F 7 140-159 20 18 20 24 23 24 F F F <--- 8 160-179 20 16 20 24 21 24 F F F <--- 9 180-199 14 14 14 18 18 18 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 20 15 20 24 20 24 F F F <--- 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 12 12 12 17 17 17 T F T <--- 18 360-379 15 15 15 20 20 20 T F T <--- 19 380-399 16 16 16 21 21 21 F F F 20 400-419 20 16 20 24 21 24 F F F <--- 21 420-439 18 18 18 23 23 23 F F F 22 440-459 18 18 18 23 23 23 F F F 23 460-479 20 18 20 24 23 24 F F F <--- 24 480-499 20 18 20 24 23 24 F F F <--- 25 500-519 20 14 20 24 18 24 F F F <--- 26 520-539 20 16 20 24 21 24 T F T <--- 27 540-559 20 18 20 24 23 24 F F F <--- 28 560-579 20 16 20 24 21 24 F F F <--- 29 580-599 20 18 20 24 23 24 F F F <--- Max gen: 20, Max flush: 24 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 25 25 25 F F F 1 020-039 12 12 12 17 17 17 F F F 2 040-059 15 15 15 20 20 20 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 18 18 18 23 23 23 F F F 5 100-119 21 21 21 25 25 25 F F F 6 120-139 18 18 18 23 23 23 F F F 7 140-159 20 20 20 24 24 24 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 20 20 20 24 24 24 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 21 21 21 25 25 25 F F F 18 360-379 21 21 21 25 25 25 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 18 18 18 23 23 23 F F F 22 440-459 18 18 18 23 23 23 F F F 23 460-479 20 20 20 24 24 24 F F F 24 480-499 20 20 20 24 24 24 F F F 25 500-519 20 20 20 24 24 24 F F F 26 520-539 21 21 21 25 25 25 F F F 27 540-559 20 20 20 24 24 24 F F F 28 560-579 20 20 20 24 24 24 F F F 29 580-599 20 20 20 24 24 24 F F F Max gen: 21, Max flush: 25 Begin loop 11 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.686735187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.698476451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.698801049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.698840787Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.698853673Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.698871621Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"Crucible 914b14b2-01ae-4749-bf0c-04fb4429096d has session id: 71935f17-8ad0-43a1-864c-bc0aed055093","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699223531Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Upstairs opts: Upstairs UUID: 914b14b2-01ae-4749-bf0c-04fb4429096d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699258127Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Crucible stats registered with UUID: 914b14b2-01ae-4749-bf0c-04fb4429096d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699274084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:28.69928759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699433347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699457217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699471874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699609397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699623483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"914b14b2-01ae-4749-bf0c-04fb4429096d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699644643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699657569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699671525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699685421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699699268Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699713374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699727541Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"ds_connection connected from Ok(127.0.0.1:47496)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699742477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"ds_connection connected from Ok(127.0.0.1:51523)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699765388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"ds_connection connected from Ok(127.0.0.1:56012)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.699807887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700464686Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700492088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700509176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700828121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700874443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[0]R flush_numbers[0..12]: [25, 17, 20, 19, 23, 25, 23, 24, 24, 18, 23, 24]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.70093538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[0]R generation[0..12]: [21, 12, 15, 14, 18, 21, 18, 20, 20, 14, 18, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700952578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700969206Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[1]R flush_numbers[0..12]: [25, 17, 20, 19, 23, 25, 23, 24, 24, 18, 23, 24]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.700984733Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[1]R generation[0..12]: [21, 12, 15, 14, 18, 21, 18, 20, 20, 14, 18, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.70099973Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701016057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[2]R flush_numbers[0..12]: [25, 17, 20, 19, 23, 25, 23, 24, 24, 18, 23, 24]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701031674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[2]R generation[0..12]: [21, 12, 15, 14, 18, 21, 18, 20, 20, 14, 18, 20]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701046781Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701063248Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701076534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701097094Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Next flush: 26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.70111023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701123616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"":"downstairs","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701139533Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701152159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"0","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701166485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"1","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701180482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"client":"2","":"downstairs","session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.70119816Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701211936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"914b14b2-01ae-4749-bf0c-04fb4429096d is now active with session: 71935f17-8ad0-43a1-864c-bc0aed055093","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701227173Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701240149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290,"session_id":"71935f17-8ad0-43a1-864c-bc0aed055093"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.701253545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} {"msg":"Activated sub_volume 914b14b2-01ae-4749-bf0c-04fb4429096d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:28.70126526Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 439 len 28672 data: 4 4 4 4 3 3 2 02/30 Flush 03/30 Flush 04/30 Write block 505 len 16384 data: 4 3 3 2 05/30 Read block 349 len 16384 06/30 Read block 219 len 40960 07/30 Write block 521 len 20480 data: 2 2 2 2 2 08/30 Read block 567 len 32768 09/30 Write block 46 len 12288 data: 2 2 2 10/30 Read block 357 len 4096 11/30 Write block 425 len 4096 data: 4 12/30 Flush 13/30 Write block 44 len 32768 data: 3 2 3 3 3 4 4 5 14/30 Read block 494 len 8192 15/30 Read block 286 len 4096 16/30 Read block 348 len 32768 17/30 Write block 582 len 20480 data: 4 3 3 3 4 18/30 Read block 391 len 4096 19/30 Read block 394 len 20480 20/30 Write block 236 len 4096 data: 4 21/30 Flush 22/30 Write block 84 len 40960 data: 2 2 4 3 3 4 5 5 4 4 23/30 Flush 24/30 Write block 574 len 4096 data: 4 25/30 Read block 277 len 32768 26/30 Write block 513 len 4096 data: 4 27/30 Write block 500 len 24576 data: 4 4 4 4 4 5 28/30 Read block 42 len 16384 29/30 Write block 491 len 40960 data: 3 2 4 4 3 3 3 4 4 5 30/30 Write block 38 len 20480 data: 4 4 4 3 4 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 26 0 26 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1001)) Some(JobId(1022)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:28.713011386Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1290} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 25 25 25 F F F 1 020-039 12 12 12 17 17 17 T F T <--- 2 040-059 22 15 22 29 20 29 T F T <--- 3 060-079 14 14 14 19 19 19 F F F 4 080-099 22 18 22 30 23 30 F F F <--- 5 100-119 21 21 21 25 25 25 F F F 6 120-139 18 18 18 23 23 23 F F F 7 140-159 20 20 20 24 24 24 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 20 22 29 24 29 F F F <--- 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 21 21 21 25 25 25 F F F 18 360-379 21 21 21 25 25 25 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 22 22 22 28 26 28 F F F <--- 22 440-459 22 22 22 26 26 26 F F F 23 460-479 20 20 20 24 24 24 F F F 24 480-499 20 20 20 24 24 24 T F T <--- 25 500-519 22 20 22 28 24 28 T F T <--- 26 520-539 22 21 22 28 25 28 F F F <--- 27 540-559 20 20 20 24 24 24 F F F 28 560-579 20 20 20 24 24 24 T F T <--- 29 580-599 22 20 22 29 24 29 F F F <--- Max gen: 22, Max flush: 30 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 25 25 25 F F F 1 020-039 23 23 23 31 31 31 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 22 22 22 30 30 30 F F F 5 100-119 21 21 21 25 25 25 F F F 6 120-139 18 18 18 23 23 23 F F F 7 140-159 20 20 20 24 24 24 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 14 14 14 18 18 18 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 15 15 15 20 20 20 F F F 17 340-359 21 21 21 25 25 25 F F F 18 360-379 21 21 21 25 25 25 F F F 19 380-399 16 16 16 21 21 21 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 22 22 22 28 28 28 F F F 22 440-459 22 22 22 26 26 26 F F F 23 460-479 20 20 20 24 24 24 F F F 24 480-499 23 23 23 31 31 31 F F F 25 500-519 23 23 23 31 31 31 F F F 26 520-539 22 22 22 28 28 28 F F F 27 540-559 20 20 20 24 24 24 F F F 28 560-579 23 23 23 31 31 31 F F F 29 580-599 22 22 22 29 29 29 F F F Max gen: 23, Max flush: 31 Begin loop 12 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.04562441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.057333199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.057700226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.057742466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.057755481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.0577739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"Crucible 03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a has session id: c7b9a9df-7ce8-4827-8214-c439d859b168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058129912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Upstairs opts: Upstairs UUID: 03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058167309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Crucible stats registered with UUID: 03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058183156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:33.058196642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058354044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058377455Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058392151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058535587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058554045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058576585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058589371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058603437Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058617254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.05863175Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058646077Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058660783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"ds_connection connected from Ok(127.0.0.1:53414)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.05871869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"ds_connection connected from Ok(127.0.0.1:35248)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.058746472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"ds_connection connected from Ok(127.0.0.1:60716)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.05876278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059458967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.05948699Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059546176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059826384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059864141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[0]R flush_numbers[0..12]: [25, 31, 31, 19, 30, 25, 23, 24, 24, 18, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.05992669Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[0]R generation[0..12]: [21, 23, 23, 14, 22, 21, 18, 20, 20, 14, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059944228Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059960755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[1]R flush_numbers[0..12]: [25, 31, 31, 19, 30, 25, 23, 24, 24, 18, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059975992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[1]R generation[0..12]: [21, 23, 23, 14, 22, 21, 18, 20, 20, 14, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.059991259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060007667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[2]R flush_numbers[0..12]: [25, 31, 31, 19, 30, 25, 23, 24, 24, 18, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060023054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[2]R generation[0..12]: [21, 23, 23, 14, 22, 21, 18, 20, 20, 14, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060041242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060057909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060071366Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060091145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Next flush: 32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060104571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060117907Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"":"downstairs","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060132093Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060144769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"0","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060159225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"1","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2026-03-14T05:01:33.060175473Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"client":"2","":"downstairs","session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.06019019Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060203866Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a is now active with session: c7b9a9df-7ce8-4827-8214-c439d859b168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060218422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060231048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298,"session_id":"c7b9a9df-7ce8-4827-8214-c439d859b168"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.060244594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} {"msg":"Activated sub_volume 03cb0023-4bfa-45bc-bfbd-7cd3f9c6e24a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:33.06025797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 507 len 4096 data: 4 02/30 Read block 69 len 12288 03/30 Write block 31 len 28672 data: 3 3 4 4 5 5 4 04/30 Flush 05/30 Write block 553 len 36864 data: 6 5 5 5 4 4 4 3 3 06/30 Read block 213 len 16384 07/30 Read block 147 len 36864 08/30 Read block 367 len 28672 09/30 Flush 10/30 Write block 326 len 16384 data: 5 5 5 5 11/30 Read block 188 len 4096 12/30 Read block 468 len 32768 13/30 Write block 453 len 20480 data: 3 2 2 3 3 14/30 Read block 10 len 16384 15/30 Write block 481 len 40960 data: 5 5 5 4 4 3 3 3 3 3 16/30 Write block 589 len 32768 data: 3 4 4 4 3 4 4 4 17/30 Write block 82 len 4096 data: 2 18/30 Read block 320 len 28672 19/30 Write block 139 len 24576 data: 4 4 2 2 3 3 20/30 Read block 590 len 8192 21/30 Read block 362 len 20480 22/30 Write block 574 len 32768 data: 5 4 4 5 5 6 6 6 23/30 Write block 371 len 28672 data: 3 3 3 4 4 4 3 24/30 Flush 25/30 Write block 353 len 8192 data: 4 5 26/30 Write block 131 len 24576 data: 3 2 2 4 5 5 27/30 Write block 194 len 12288 data: 3 3 3 28/30 Read block 227 len 36864 29/30 Write block 360 len 12288 data: 3 2 2 30/30 Write block 374 len 28672 data: 5 5 5 4 2 2 2 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:33.072703085Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1298} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 25 25 25 F F F 1 020-039 24 24 23 32 32 31 F F F <--- 2 040-059 23 23 23 31 31 31 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 24 24 22 34 34 30 F F F <--- 5 100-119 21 21 21 25 25 25 F F F 6 120-139 24 24 18 34 34 23 T T F <--- 7 140-159 24 24 20 34 34 24 F F F <--- 8 160-179 20 20 20 24 24 24 F F F 9 180-199 14 14 14 18 18 18 T T F <--- 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 24 24 15 34 34 20 F F F <--- 17 340-359 21 21 21 25 25 25 T T F <--- 18 360-379 24 24 21 34 34 25 T T F <--- 19 380-399 16 16 16 21 21 21 T T F <--- 20 400-419 20 20 20 24 24 24 F F F 21 420-439 22 22 22 28 28 28 F F F 22 440-459 24 24 22 34 34 26 F F F <--- 23 460-479 20 20 20 24 24 24 F F F 24 480-499 24 24 23 34 34 31 F F F <--- 25 500-519 24 24 23 32 32 31 F F F <--- 26 520-539 22 22 22 28 28 28 F F F 27 540-559 24 24 20 33 33 24 F F F <--- 28 560-579 24 24 23 34 34 31 F F F <--- 29 580-599 24 24 22 34 34 29 F F F <--- Max gen: 24, Max flush: 34 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 25 25 25 F F F 1 020-039 24 24 24 32 32 32 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 14 14 14 19 19 19 F F F 4 080-099 24 24 24 34 34 34 F F F 5 100-119 21 21 21 25 25 25 F F F 6 120-139 25 25 25 35 35 35 F F F 7 140-159 24 24 24 34 34 34 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 25 25 25 35 35 35 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 18 18 18 23 23 23 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 24 24 24 34 34 34 F F F 17 340-359 25 25 25 35 35 35 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 25 25 25 35 35 35 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 22 22 22 28 28 28 F F F 22 440-459 24 24 24 34 34 34 F F F 23 460-479 20 20 20 24 24 24 F F F 24 480-499 24 24 24 34 34 34 F F F 25 500-519 24 24 24 32 32 32 F F F 26 520-539 22 22 22 28 28 28 F F F 27 540-559 24 24 24 33 33 33 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 25, Max flush: 35 Begin loop 13 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.399816926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.411464867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.411780501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.411819159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.411831574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.411849452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"Crucible 9f27749d-4f9e-4160-82b8-0e06b8305103 has session id: 024df554-386b-4293-888c-641cf590a9fa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412264251Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Upstairs opts: Upstairs UUID: 9f27749d-4f9e-4160-82b8-0e06b8305103, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412299257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Crucible stats registered with UUID: 9f27749d-4f9e-4160-82b8-0e06b8305103","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412314734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:37.41232813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412493425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412528521Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412544118Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412639842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.41265641Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"9f27749d-4f9e-4160-82b8-0e06b8305103 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412686893Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.41270082Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412715116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412729053Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412743049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412757496Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412771682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:46825)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412796753Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:45566)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.41281174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:44906)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.412933646Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.413588704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.413617507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.413636486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.413959623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414007555Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[0]R flush_numbers[0..12]: [25, 32, 31, 19, 34, 25, 35, 34, 24, 35, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414091193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[0]R generation[0..12]: [21, 24, 23, 14, 24, 21, 25, 24, 20, 25, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.4141081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414125138Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[1]R flush_numbers[0..12]: [25, 32, 31, 19, 34, 25, 35, 34, 24, 35, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414140275Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[1]R generation[0..12]: [21, 24, 23, 14, 24, 21, 25, 24, 20, 25, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414155562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414171599Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[2]R flush_numbers[0..12]: [25, 32, 31, 19, 34, 25, 35, 34, 24, 35, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414187076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[2]R generation[0..12]: [21, 24, 23, 14, 24, 21, 25, 24, 20, 25, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414202653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414218891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414232327Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414250785Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Next flush: 36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414263901Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414277247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"":"downstairs","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414291324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414306351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"0","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414320157Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"1","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414334283Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"client":"2","":"downstairs","session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :30,"time":"2026-03-14T05:01:37.414351461Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414364947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"9f27749d-4f9e-4160-82b8-0e06b8305103 is now active with session: 024df554-386b-4293-888c-641cf590a9fa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414380034Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.4143927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306,"session_id":"024df554-386b-4293-888c-641cf590a9fa"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414406266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} {"msg":"Activated sub_volume 9f27749d-4f9e-4160-82b8-0e06b8305103","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:37.414417531Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 150 len 40960 data: 3 3 2 2 4 4 4 4 5 5 02/30 Write block 564 len 36864 data: 2 3 3 3 3 5 4 3 4 03/30 Read block 412 len 4096 04/30 Read block 55 len 32768 05/30 Flush 06/30 Write block 189 len 40960 data: 3 2 2 2 2 4 4 4 3 3 07/30 Write block 285 len 32768 data: 5 6 5 5 6 4 4 4 08/30 Read block 32 len 12288 09/30 Write block 72 len 40960 data: 3 2 2 3 2 3 2 2 2 2 10/30 Write block 382 len 32768 data: 2 2 2 2 2 2 2 2 11/30 Read block 136 len 16384 12/30 Write block 568 len 40960 data: 4 6 5 4 5 3 6 5 5 6 13/30 Write block 122 len 40960 data: 5 4 4 3 2 3 3 3 3 4 14/30 Read block 435 len 16384 15/30 Write block 8 len 24576 data: 3 4 3 3 3 3 16/30 Read block 61 len 40960 17/30 Write block 473 len 20480 data: 2 2 2 2 3 18/30 Write block 26 len 4096 data: 4 19/30 Read block 78 len 12288 20/30 Write block 348 len 20480 data: 3 3 3 5 4 21/30 Read block 38 len 4096 22/30 Flush 23/30 Write block 435 len 24576 data: 3 3 2 4 5 5 24/30 Read block 514 len 28672 25/30 Write block 78 len 36864 data: 3 3 3 3 3 2 3 3 5 26/30 Write block 489 len 24576 data: 4 4 4 3 5 5 27/30 Read block 78 len 8192 28/30 Write block 522 len 8192 data: 3 3 29/30 Read block 573 len 4096 30/30 Write block 92 len 32768 data: 5 5 4 3 2 2 2 2 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 10 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Write 10 Done Done Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 21 1 23 Done 24 4 24 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1004)) Some(JobId(1021)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:37.428083101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1306} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 26 21 26 37 25 37 F F F <--- 1 020-039 26 24 26 37 32 37 F F F <--- 2 040-059 23 23 23 31 31 31 F F F 3 060-079 26 14 26 37 19 37 T T T <--- 4 080-099 26 24 26 37 34 37 T T T <--- 5 100-119 21 21 21 25 25 25 F F F 6 120-139 26 25 26 37 35 37 F F F <--- 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 26 25 26 37 35 37 F T F <--- 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 26 18 26 37 23 37 F T F <--- 15 300-319 17 17 17 22 22 22 F F F 16 320-339 24 24 24 34 34 34 F F F 17 340-359 26 25 26 37 35 37 F F F <--- 18 360-379 25 25 25 35 35 35 F F F 19 380-399 26 25 26 37 35 37 F F F <--- 20 400-419 20 20 20 24 24 24 F F F 21 420-439 22 22 22 28 28 28 T F T <--- 22 440-459 24 24 24 34 34 34 T F T <--- 23 460-479 26 20 26 37 24 37 F F F <--- 24 480-499 24 24 24 34 34 34 T F T <--- 25 500-519 24 24 24 32 32 32 F F F 26 520-539 22 22 22 28 28 28 T F T <--- 27 540-559 24 24 24 33 33 33 F F F 28 560-579 26 26 26 37 36 37 F F F <--- 29 580-599 24 24 24 34 34 34 F F F Max gen: 26, Max flush: 37 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 26 26 26 37 37 37 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 27 27 27 38 38 38 F F F 4 080-099 27 27 27 38 38 38 F F F 5 100-119 21 21 21 25 25 25 F F F 6 120-139 26 26 26 37 37 37 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 26 26 26 37 37 37 F F F 10 200-219 18 18 18 23 23 23 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 18 18 18 23 23 23 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 17 17 17 22 22 22 F F F 16 320-339 24 24 24 34 34 34 F F F 17 340-359 26 26 26 37 37 37 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 26 26 26 37 37 37 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 27 27 27 38 38 38 F F F 22 440-459 27 27 27 38 38 38 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 27 27 27 38 38 38 F F F 25 500-519 24 24 24 32 32 32 F F F 26 520-539 27 27 27 38 38 38 F F F 27 540-559 24 24 24 33 33 33 F F F 28 560-579 26 26 26 37 37 37 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 27, Max flush: 38 Begin loop 14 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.759053399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.770772913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771101332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.77114038Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771152605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771170494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"Crucible 4072c648-4bc9-436d-b5b4-35a9d4a4fa25 has session id: 6809a49b-7bf6-4037-94fa-db40f53dc6e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771519853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Upstairs opts: Upstairs UUID: 4072c648-4bc9-436d-b5b4-35a9d4a4fa25, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771582221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Crucible stats registered with UUID: 4072c648-4bc9-436d-b5b4-35a9d4a4fa25","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771598638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:41.771611914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771764764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771789105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771804222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771904548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771921135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"4072c648-4bc9-436d-b5b4-35a9d4a4fa25 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771938503Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771951549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771965836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.771979762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772027684Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772043001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:40139)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772057807Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772071744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:41889)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.77215041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36373)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772168488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772757806Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772800505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.772956637Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773122562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.77316101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[0]R flush_numbers[0..12]: [37, 37, 31, 38, 38, 25, 37, 36, 24, 37, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773314169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[0]R generation[0..12]: [26, 26, 23, 27, 27, 21, 26, 26, 20, 26, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773331297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773347925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[1]R flush_numbers[0..12]: [37, 37, 31, 38, 38, 25, 37, 36, 24, 37, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773363172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[1]R generation[0..12]: [26, 26, 23, 27, 27, 21, 26, 26, 20, 26, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773378398Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773394846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[2]R flush_numbers[0..12]: [37, 37, 31, 38, 38, 25, 37, 36, 24, 37, 23, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773409983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[2]R generation[0..12]: [26, 26, 23, 27, 27, 21, 26, 26, 20, 26, 18, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773428531Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773444859Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773458455Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773478444Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Next flush: 39","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.77349184Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773505576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"":"downstairs","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773519913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773532289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"0","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773546375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"1","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:01:41.773560001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"client":"2","":"downstairs","session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773575838Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773589224Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"4072c648-4bc9-436d-b5b4-35a9d4a4fa25 is now active with session: 6809a49b-7bf6-4037-94fa-db40f53dc6e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773604381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773617057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314,"session_id":"6809a49b-7bf6-4037-94fa-db40f53dc6e6"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773630333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} {"msg":"Activated sub_volume 4072c648-4bc9-436d-b5b4-35a9d4a4fa25","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:41.773643549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 263 len 40960 data: 3 3 3 4 4 4 4 4 4 4 03/30 Read block 182 len 16384 04/30 Write block 319 len 32768 data: 6 5 5 5 4 4 4 6 05/30 Write block 422 len 40960 data: 3 3 4 5 4 4 3 3 3 3 06/30 Read block 83 len 24576 07/30 Read block 538 len 40960 08/30 Read block 437 len 8192 09/30 Write block 308 len 20480 data: 2 2 2 2 2 10/30 Read block 298 len 8192 11/30 Write block 133 len 16384 data: 3 5 6 6 12/30 Read block 479 len 12288 13/30 Flush 14/30 Write block 246 len 40960 data: 3 3 2 2 2 2 3 3 4 4 15/30 Write block 568 len 32768 data: 5 7 6 5 6 4 7 6 16/30 Write block 212 len 24576 data: 4 2 2 2 3 3 17/30 Write block 73 len 36864 data: 3 3 4 3 4 4 4 4 4 18/30 Write block 134 len 24576 data: 6 7 7 5 5 5 19/30 Write block 116 len 8192 data: 3 3 20/30 Read block 285 len 24576 21/30 Read block 541 len 20480 22/30 Read block 361 len 24576 23/30 Read block 449 len 8192 24/30 Read block 484 len 40960 25/30 Flush 26/30 Write block 388 len 28672 data: 3 3 2 2 2 2 3 27/30 Read block 83 len 4096 28/30 Write block 553 len 28672 data: 7 6 6 6 5 5 5 29/30 Write block 432 len 32768 data: 3 3 4 4 4 3 5 6 30/30 Write block 194 len 16384 data: 5 5 5 4 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Done Sent false 1001 Acked Write 10 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:41.786012689Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1314} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 26 26 26 37 37 37 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 28 28 27 41 41 38 F F F <--- 4 080-099 28 28 27 41 41 38 F F F <--- 5 100-119 28 28 21 41 41 25 F F F <--- 6 120-139 28 28 26 41 41 37 F F F <--- 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 26 26 26 37 37 37 T T F <--- 10 200-219 28 28 18 41 41 23 F F F <--- 11 220-239 22 22 22 29 29 29 F F F 12 240-259 28 28 14 41 41 19 F F F <--- 13 260-279 28 28 18 40 40 23 F F F <--- 14 280-299 26 26 26 37 37 37 F F F 15 300-319 28 28 17 40 40 22 F F F <--- 16 320-339 28 28 24 40 40 34 F F F <--- 17 340-359 26 26 26 37 37 37 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 26 26 26 37 37 37 T T F <--- 20 400-419 20 20 20 24 24 24 F F F 21 420-439 28 28 27 40 40 38 T T F <--- 22 440-459 27 27 27 38 38 38 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 27 27 27 38 38 38 F F F 25 500-519 24 24 24 32 32 32 F F F 26 520-539 27 27 27 38 38 38 F F F 27 540-559 24 24 24 33 33 33 T T F <--- 28 560-579 28 28 26 41 41 37 F F F <--- 29 580-599 24 24 24 34 34 34 F F F Max gen: 28, Max flush: 41 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 26 26 26 37 37 37 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 28 28 28 41 41 41 F F F 4 080-099 28 28 28 41 41 41 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 28 28 28 41 41 41 F F F 11 220-239 22 22 22 29 29 29 F F F 12 240-259 28 28 28 41 41 41 F F F 13 260-279 28 28 28 40 40 40 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 28 28 28 40 40 40 F F F 16 320-339 28 28 28 40 40 40 F F F 17 340-359 26 26 26 37 37 37 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 29 29 29 42 42 42 F F F 20 400-419 20 20 20 24 24 24 F F F 21 420-439 29 29 29 42 42 42 F F F 22 440-459 27 27 27 38 38 38 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 27 27 27 38 38 38 F F F 25 500-519 24 24 24 32 32 32 F F F 26 520-539 27 27 27 38 38 38 F F F 27 540-559 29 29 29 42 42 42 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 29, Max flush: 42 Begin loop 15 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.115876271Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:46.126500526Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.126547577Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.13692271Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137229679Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137270458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137282443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137300061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"Crucible 3da464fc-d116-4290-9852-6a3585d2ab3c has session id: d28c7d38-e737-4698-b723-79999cc495bf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137631562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Upstairs opts: Upstairs UUID: 3da464fc-d116-4290-9852-6a3585d2ab3c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.13767022Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Crucible stats registered with UUID: 3da464fc-d116-4290-9852-6a3585d2ab3c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137685027Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:46.137697962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137865279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137888049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.137905257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138030824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138046431Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"3da464fc-d116-4290-9852-6a3585d2ab3c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138067591Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138080537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138094563Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138108539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"2","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138122095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138136342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138150789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"ds_connection connected from Ok(127.0.0.1:53289)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138206274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"ds_connection connected from Ok(127.0.0.1:64973)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138299506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"ds_connection connected from Ok(127.0.0.1:60210)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138365966Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138866454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"2","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.138894397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139012151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139267927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139307705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[0]R flush_numbers[0..12]: [37, 37, 31, 41, 41, 41, 41, 36, 24, 42, 41, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139369393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[0]R generation[0..12]: [26, 26, 23, 28, 28, 28, 28, 26, 20, 29, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139386591Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139403259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[1]R flush_numbers[0..12]: [37, 37, 31, 41, 41, 41, 41, 36, 24, 42, 41, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139418866Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[1]R generation[0..12]: [26, 26, 23, 28, 28, 28, 28, 26, 20, 29, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139434063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.13945055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[2]R flush_numbers[0..12]: [37, 37, 31, 41, 41, 41, 41, 36, 24, 42, 41, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139466617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[2]R generation[0..12]: [26, 26, 23, 28, 28, 28, 28, 26, 20, 29, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139482435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139498922Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:01:46.139512468Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139530547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Next flush: 43","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139543933Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.1395595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"":"downstairs","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139573906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139586722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"0","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139601009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"1","":"downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139615365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"client":"2","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139632813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139646439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"3da464fc-d116-4290-9852-6a3585d2ab3c is now active with session: d28c7d38-e737-4698-b723-79999cc495bf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139661436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139674282Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322,"session_id":"d28c7d38-e737-4698-b723-79999cc495bf"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139687798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} {"msg":"Activated sub_volume 3da464fc-d116-4290-9852-6a3585d2ab3c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:46.139699143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 215 len 12288 data: 3 4 4 02/30 Write block 246 len 28672 data: 4 4 3 3 3 3 4 03/30 Read block 205 len 32768 04/30 Read block 271 len 32768 05/30 Read block 382 len 32768 06/30 Read block 500 len 12288 07/30 Write block 318 len 36864 data: 5 7 6 6 6 5 5 5 7 08/30 Read block 132 len 8192 09/30 Read block 577 len 16384 10/30 Write block 226 len 40960 data: 4 4 4 4 5 5 4 3 4 4 11/30 Write block 494 len 24576 data: 6 4 4 4 5 5 12/30 Write block 386 len 24576 data: 3 3 4 4 3 3 13/30 Read block 593 len 20480 14/30 Flush 15/30 Write block 492 len 36864 data: 4 6 7 5 5 5 6 6 6 16/30 Read block 490 len 28672 17/30 Write block 407 len 24576 data: 4 4 3 3 3 3 18/30 Write block 64 len 24576 data: 5 5 6 6 5 4 19/30 Flush 20/30 Write block 452 len 32768 data: 3 4 3 3 4 4 3 2 21/30 Read block 431 len 28672 22/30 Write block 339 len 24576 data: 4 4 4 3 4 3 23/30 Read block 158 len 16384 24/30 Read block 469 len 24576 25/30 Flush 26/30 Flush 27/30 Write block 12 len 20480 data: 4 4 3 3 3 28/30 Read block 386 len 40960 29/30 Read block 204 len 12288 30/30 Read block 213 len 8192 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 3 30 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:46.152845617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1322} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 26 26 26 37 37 37 F T T <--- 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 28 30 30 41 44 44 F F F <--- 4 080-099 28 28 28 41 41 41 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 28 30 30 41 43 43 T F F <--- 11 220-239 22 30 30 29 43 43 F F F <--- 12 240-259 28 30 30 41 43 43 T F F <--- 13 260-279 28 28 28 40 40 40 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 28 30 30 40 43 43 F F F <--- 16 320-339 28 30 30 40 45 45 F F F <--- 17 340-359 26 30 30 37 45 45 F F F <--- 18 360-379 25 25 25 35 35 35 F F F 19 380-399 29 30 30 42 43 43 F F F <--- 20 400-419 20 30 30 24 44 44 F F F <--- 21 420-439 29 29 29 42 42 42 F F F 22 440-459 27 30 30 38 45 45 F F F <--- 23 460-479 26 26 26 37 37 37 F F F 24 480-499 27 30 30 38 44 44 F F F <--- 25 500-519 24 30 30 32 44 44 F F F <--- 26 520-539 27 27 27 38 38 38 F F F 27 540-559 29 29 29 42 42 42 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 30, Max flush: 45 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 46 46 46 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 30 30 30 44 44 44 F F F 4 080-099 28 28 28 41 41 41 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 30 30 30 43 43 43 F F F 11 220-239 30 30 30 43 43 43 F F F 12 240-259 30 30 30 43 43 43 F F F 13 260-279 28 28 28 40 40 40 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 30 30 30 43 43 43 F F F 16 320-339 30 30 30 45 45 45 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 30 30 30 43 43 43 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 29 29 29 42 42 42 F F F 22 440-459 30 30 30 45 45 45 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 30 30 30 44 44 44 F F F 25 500-519 30 30 30 44 44 44 F F F 26 520-539 27 27 27 38 38 38 F F F 27 540-559 29 29 29 42 42 42 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 31, Max flush: 46 Begin loop 16 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.487058649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.498840462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499152064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499190921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499202867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499220575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"Crucible 9df9163a-cd22-497e-ba32-86164579dfbc has session id: 4aef9a69-4fab-41ed-aa2e-d7758f5f3959","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.49956148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Upstairs opts: Upstairs UUID: 9df9163a-cd22-497e-ba32-86164579dfbc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499596456Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Crucible stats registered with UUID: 9df9163a-cd22-497e-ba32-86164579dfbc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499612173Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:50.499625629Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499794616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499817466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499832223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499965754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.499982071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"9df9163a-cd22-497e-ba32-86164579dfbc active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500003321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500016097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500029913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500043829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500057696Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500115172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500142895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"ds_connection connected from Ok(127.0.0.1:44343)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500158752Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"ds_connection connected from Ok(127.0.0.1:52882)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500173889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"ds_connection connected from Ok(127.0.0.1:48873)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500188735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500904921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500949061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.500966129Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501277711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501308115Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[0]R flush_numbers[0..12]: [46, 37, 31, 44, 41, 41, 41, 36, 24, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501379087Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[0]R generation[0..12]: [31, 26, 23, 30, 28, 28, 28, 26, 20, 29, 30, 30]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501396645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501413463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[1]R flush_numbers[0..12]: [46, 37, 31, 44, 41, 41, 41, 36, 24, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.50142882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[1]R generation[0..12]: [31, 26, 23, 30, 28, 28, 28, 26, 20, 29, 30, 30]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501444227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501463566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[2]R flush_numbers[0..12]: [46, 37, 31, 44, 41, 41, 41, 36, 24, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501479653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[2]R generation[0..12]: [31, 26, 23, 30, 28, 28, 28, 26, 20, 29, 30, 30]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.50149483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501511017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:01:50.501524713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501544723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501557929Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501571615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"":"downstairs","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501585821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501598507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"0","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501612754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"1","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.50162678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"client":"2","":"downstairs","session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501642667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501656383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"9df9163a-cd22-497e-ba32-86164579dfbc is now active with session: 4aef9a69-4fab-41ed-aa2e-d7758f5f3959","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.50167132Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501685627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330,"session_id":"4aef9a69-4fab-41ed-aa2e-d7758f5f3959"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501698933Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} {"msg":"Activated sub_volume 9df9163a-cd22-497e-ba32-86164579dfbc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:50.501710848Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 210 len 32768 data: 4 5 5 3 3 4 5 5 02/30 Flush 03/30 Write block 494 len 28672 data: 8 6 6 6 7 7 7 04/30 Read block 63 len 24576 05/30 Read block 63 len 28672 06/30 Write block 158 len 8192 data: 6 6 07/30 Write block 10 len 32768 data: 4 4 5 5 4 4 4 3 08/30 Read block 219 len 12288 09/30 Write block 213 len 40960 data: 4 4 5 6 6 3 3 3 3 3 10/30 Read block 386 len 32768 11/30 Write block 580 len 40960 data: 7 7 5 4 4 4 5 3 3 4 12/30 Write block 272 len 20480 data: 5 5 4 3 3 13/30 Read block 539 len 24576 14/30 Write block 66 len 24576 data: 7 7 6 5 4 3 15/30 Write block 551 len 20480 data: 5 5 8 7 7 16/30 Read block 350 len 36864 17/30 Read block 410 len 20480 18/30 Read block 187 len 8192 19/30 Write block 514 len 28672 data: 5 4 4 3 3 2 2 20/30 Flush 21/30 Write block 307 len 40960 data: 3 3 3 3 3 3 2 3 4 5 22/30 Read block 558 len 32768 23/30 Read block 452 len 16384 24/30 Read block 3 len 4096 25/30 Write block 213 len 8192 data: 5 5 26/30 Write block 448 len 12288 data: 3 3 3 27/30 Read block 263 len 28672 28/30 Write block 540 len 4096 data: 4 29/30 Read block 105 len 36864 30/30 Read block 135 len 28672 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 7 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Read 7 Done Done Done false 1005 Acked Write 2 Done Done Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 3 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 5 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 2 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 1 25 Done 28 4 27 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1001)) Some(JobId(1019)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:50.515100702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1330} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 31 32 48 46 48 F F F <--- 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 32 30 32 48 44 48 F F F <--- 4 080-099 28 28 28 41 41 41 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 32 26 32 48 36 48 F T F <--- 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 32 32 32 48 47 48 T F T <--- 11 220-239 32 30 32 48 43 48 F F F <--- 12 240-259 30 30 30 43 43 43 F F F 13 260-279 32 28 32 48 40 48 F F F <--- 14 280-299 26 26 26 37 37 37 F F F 15 300-319 30 30 30 43 43 43 T F T <--- 16 320-339 30 30 30 45 45 45 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 30 30 30 43 43 43 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 29 29 29 42 42 42 F F F 22 440-459 30 30 30 45 45 45 T F T <--- 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 30 32 48 44 48 F T F <--- 25 500-519 32 30 32 48 44 48 F T F <--- 26 520-539 32 27 32 48 38 48 F F F <--- 27 540-559 32 29 32 48 42 48 T F T <--- 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 24 32 48 34 48 F F F <--- Max gen: 32, Max flush: 48 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 F F F 3 060-079 32 32 32 48 48 48 F F F 4 080-099 28 28 28 41 41 41 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 32 32 32 48 48 48 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 32 32 48 48 48 F F F 12 240-259 30 30 30 43 43 43 F F F 13 260-279 32 32 32 48 48 48 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 33 33 33 49 49 49 F F F 16 320-339 30 30 30 45 45 45 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 25 25 25 35 35 35 F F F 19 380-399 30 30 30 43 43 43 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 29 29 29 42 42 42 F F F 22 440-459 33 33 33 49 49 49 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 32 32 32 48 48 48 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F F F Max gen: 33, Max flush: 49 Begin loop 17 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.840752057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.8524052Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.852719353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.852759312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.852771517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.852789185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"Crucible 00e6cd3a-c6ae-49e5-97d7-27717220a7aa has session id: f7f3aae9-7817-4a55-8ffa-fda2337e88ce","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853124908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Upstairs opts: Upstairs UUID: 00e6cd3a-c6ae-49e5-97d7-27717220a7aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853159524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Crucible stats registered with UUID: 00e6cd3a-c6ae-49e5-97d7-27717220a7aa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853175161Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:54.853188637Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853388758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853423474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853439031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853527091Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853541628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"00e6cd3a-c6ae-49e5-97d7-27717220a7aa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853560876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853573722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853587779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853601585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853615401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853629838Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853644004Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:58591)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853669846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:50445)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.853685113Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:39395)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.85369999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854556159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854577399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854593536Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854947728Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854963275Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[0]R flush_numbers[0..12]: [48, 37, 31, 48, 41, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.854989587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[0]R generation[0..12]: [32, 26, 23, 32, 28, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855005264Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855021791Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[1]R flush_numbers[0..12]: [48, 37, 31, 48, 41, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855037198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[1]R generation[0..12]: [32, 26, 23, 32, 28, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855052425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855068433Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[2]R flush_numbers[0..12]: [48, 37, 31, 48, 41, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855083389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[2]R generation[0..12]: [32, 26, 23, 32, 28, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855098346Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855114634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.85512847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855142736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Next flush: 50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855155882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855169038Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"":"downstairs","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855191499Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855204294Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"0","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855218121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"1","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855232067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"client":"2","":"downstairs","session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855245963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855259289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"00e6cd3a-c6ae-49e5-97d7-27717220a7aa is now active with session: f7f3aae9-7817-4a55-8ffa-fda2337e88ce","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855274116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:01:54.855286432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338,"session_id":"f7f3aae9-7817-4a55-8ffa-fda2337e88ce"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855302349Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} {"msg":"Activated sub_volume 00e6cd3a-c6ae-49e5-97d7-27717220a7aa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:54.855313794Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 436 len 12288 data: 5 4 6 02/30 Read block 248 len 24576 03/30 Write block 283 len 32768 data: 3 4 6 7 6 6 7 5 04/30 Read block 446 len 40960 05/30 Read block 29 len 8192 06/30 Write block 368 len 28672 data: 4 4 4 4 4 4 6 07/30 Read block 561 len 4096 08/30 Read block 174 len 20480 09/30 Flush 10/30 Write block 379 len 24576 data: 3 3 2 3 3 3 11/30 Flush 12/30 Write block 451 len 4096 data: 3 13/30 Write block 528 len 24576 data: 2 2 2 3 4 4 14/30 Write block 295 len 36864 data: 2 2 2 2 2 2 2 2 2 15/30 Read block 590 len 4096 16/30 Write block 52 len 36864 data: 4 5 6 5 4 3 4 4 4 17/30 Read block 399 len 40960 18/30 Write block 85 len 40960 data: 4 6 4 4 5 6 6 6 6 5 19/30 Read block 355 len 12288 20/30 Read block 66 len 12288 21/30 Read block 562 len 4096 22/30 Read block 524 len 4096 23/30 Read block 39 len 36864 24/30 Read block 399 len 8192 25/30 Read block 308 len 32768 26/30 Write block 424 len 40960 data: 5 6 5 5 4 4 4 4 4 4 27/30 Read block 315 len 40960 28/30 Write block 367 len 24576 data: 4 5 5 5 5 5 29/30 Read block 9 len 4096 30/30 Write block 248 len 12288 data: 4 4 4 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 6 Done Done Sent false 1002 Acked Write 8 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:54.86831114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1338} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 23 23 23 31 31 31 T T F <--- 3 060-079 32 32 32 48 48 48 T T F <--- 4 080-099 28 28 28 41 41 41 T T F <--- 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 32 32 32 48 48 48 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 32 32 48 48 48 F F F 12 240-259 30 30 30 43 43 43 T T F <--- 13 260-279 32 32 32 48 48 48 F F F 14 280-299 34 34 26 50 50 37 T T F <--- 15 300-319 33 33 33 49 49 49 T T F <--- 16 320-339 30 30 30 45 45 45 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 34 34 25 51 51 35 T T F <--- 19 380-399 34 34 30 51 51 43 F F F <--- 20 400-419 30 30 30 44 44 44 F F F 21 420-439 34 34 29 50 50 42 T T T <--- 22 440-459 33 33 33 49 49 49 T T F <--- 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 32 32 32 48 48 48 T T F <--- 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F F F Max gen: 34, Max flush: 51 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 35 35 35 52 52 52 F F F 3 060-079 35 35 35 52 52 52 F F F 4 080-099 35 35 35 52 52 52 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F F F 7 140-159 32 32 32 48 48 48 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 32 32 48 48 48 F F F 12 240-259 35 35 35 52 52 52 F F F 13 260-279 32 32 32 48 48 48 F F F 14 280-299 35 35 35 52 52 52 F F F 15 300-319 35 35 35 52 52 52 F F F 16 320-339 30 30 30 45 45 45 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 35 35 35 52 52 52 F F F 19 380-399 34 34 34 51 51 51 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 35 35 35 52 52 52 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F F F Max gen: 35, Max flush: 52 Begin loop 18 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.192700822Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:59.203323527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.203362555Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.213765439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.21409517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214134718Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214150895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214168953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"Crucible 41e29b15-6fa6-44ae-8bb9-c44a1099885e has session id: 09b99d6c-b567-4bed-a814-0e95d518f32a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214478954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Upstairs opts: Upstairs UUID: 41e29b15-6fa6-44ae-8bb9-c44a1099885e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.21451319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Crucible stats registered with UUID: 41e29b15-6fa6-44ae-8bb9-c44a1099885e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214528767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:59.214542133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214673503Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214698124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214713091Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214832295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214846682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"41e29b15-6fa6-44ae-8bb9-c44a1099885e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.21486449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214877226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214891192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214905278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214918774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214936252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214950659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46845)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.214999181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:36420)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.215027084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:37892)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.215043211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.21583124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.215859152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.215883864Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216212063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216247259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[0]R flush_numbers[0..12]: [48, 37, 52, 52, 52, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216318051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[0]R generation[0..12]: [32, 26, 35, 35, 35, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216335499Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216352297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[1]R flush_numbers[0..12]: [48, 37, 52, 52, 52, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216367774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[1]R generation[0..12]: [32, 26, 35, 35, 35, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216386042Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.21640254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[2]R flush_numbers[0..12]: [48, 37, 52, 52, 52, 41, 41, 48, 24, 42, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216417877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[2]R generation[0..12]: [32, 26, 35, 35, 35, 28, 28, 32, 20, 29, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216432854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216449041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216462567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":Wait for a query_work_queue command to finish before sending IO 1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216483567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Next flush: 53","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216497113Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216510669Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"":"downstairs","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216524885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216537681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"0","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216551648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"1","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216565544Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"client":"2","":"downstairs","session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 time":"2026-03-14T05:01:59.21657981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216595708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"41e29b15-6fa6-44ae-8bb9-c44a1099885e is now active with session: 09b99d6c-b567-4bed-a814-0e95d518f32a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216612605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216625201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346,"session_id":"09b99d6c-b567-4bed-a814-0e95d518f32a"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216638827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} {"msg":"Activated sub_volume 41e29b15-6fa6-44ae-8bb9-c44a1099885e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:59.216650202Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 77 len 40960 data: 5 5 5 5 5 4 3 4 5 7 02/30 Read block 384 len 40960 03/30 Read block 390 len 8192 04/30 Read block 519 len 4096 05/30 Read block 218 len 24576 06/30 Read block 241 len 16384 07/30 Read block 392 len 32768 08/30 Write block 292 len 24576 data: 5 4 3 3 3 3 09/30 Read block 541 len 36864 10/30 Flush 11/30 Write block 274 len 20480 data: 5 4 4 4 3 12/30 Read block 56 len 20480 13/30 Write block 368 len 8192 data: 6 6 14/30 Write block 392 len 4096 data: 3 15/30 Read block 322 len 32768 16/30 Read block 72 len 36864 17/30 Write block 87 len 36864 data: 5 5 6 7 7 7 7 6 4 18/30 Write block 348 len 24576 data: 4 4 4 6 5 5 19/30 Read block 109 len 12288 20/30 Read block 512 len 12288 21/30 Flush 22/30 Write block 316 len 20480 data: 6 5 6 8 7 23/30 Write block 189 len 4096 data: 4 24/30 Read block 24 len 32768 25/30 Read block 118 len 28672 26/30 Read block 99 len 36864 27/30 Read block 520 len 16384 28/30 Write block 126 len 28672 data: 3 4 4 4 4 5 3 29/30 Read block 146 len 24576 30/30 Write block 249 len 32768 data: 5 5 4 5 4 5 5 4 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 10 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:59.229569162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1346} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 35 35 35 52 52 52 F F F 3 060-079 35 36 36 52 53 53 T F F <--- 4 080-099 35 36 36 52 54 54 T F F <--- 5 100-119 28 28 28 41 41 41 F F F 6 120-139 28 28 28 41 41 41 F T T <--- 7 140-159 32 32 32 48 48 48 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 29 29 29 42 42 42 F T T <--- 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 32 32 48 48 48 F F F 12 240-259 35 35 35 52 52 52 F T T <--- 13 260-279 32 36 36 48 54 54 F F F <--- 14 280-299 35 36 36 52 53 53 F F F <--- 15 300-319 35 35 35 52 52 52 F T T <--- 16 320-339 30 30 30 45 45 45 F T T <--- 17 340-359 30 36 36 45 54 54 F F F <--- 18 360-379 35 36 36 52 54 54 F F F <--- 19 380-399 34 36 36 51 54 54 F F F <--- 20 400-419 30 30 30 44 44 44 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 35 35 35 52 52 52 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F F F Max gen: 36, Max flush: 54 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 26 26 37 37 37 F F F 2 040-059 35 35 35 52 52 52 F F F 3 060-079 36 36 36 53 53 53 F F F 4 080-099 36 36 36 54 54 54 F F F 5 100-119 28 28 28 41 41 41 F F F 6 120-139 37 37 37 55 55 55 F F F 7 140-159 32 32 32 48 48 48 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 37 37 37 55 55 55 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 32 32 48 48 48 F F F 12 240-259 37 37 37 55 55 55 F F F 13 260-279 36 36 36 54 54 54 F F F 14 280-299 36 36 36 53 53 53 F F F 15 300-319 37 37 37 55 55 55 F F F 16 320-339 37 37 37 55 55 55 F F F 17 340-359 36 36 36 54 54 54 F F F 18 360-379 36 36 36 54 54 54 F F F 19 380-399 36 36 36 54 54 54 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 35 35 35 52 52 52 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F F F Max gen: 37, Max flush: 55 Begin loop 19 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.564814395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:03.575402164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.575457279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.585633021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.585918661Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.585958319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.585970454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.585988612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"Crucible 34a297ad-d848-4667-83e4-29a969a32aaa has session id: d1df33b7-e15e-438f-b63b-d6f0f0735ed9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586355019Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Upstairs opts: Upstairs UUID: 34a297ad-d848-4667-83e4-29a969a32aaa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586389405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Crucible stats registered with UUID: 34a297ad-d848-4667-83e4-29a969a32aaa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586405492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:03.58642232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.58657607Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586610846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586627003Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586734722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586750229Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"34a297ad-d848-4667-83e4-29a969a32aaa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586771459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586784355Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586798041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586811957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586855377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586880428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586895505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"ds_connection connected from Ok(127.0.0.1:63238)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586910812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"ds_connection connected from Ok(127.0.0.1:33875)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.586937394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"ds_connection connected from Ok(127.0.0.1:41945)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.587049315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.587621576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.587661274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.587677811Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.587967703Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588006741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[0]R flush_numbers[0..12]: [48, 37, 52, 53, 54, 41, 55, 48, 24, 55, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588067138Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[0]R generation[0..12]: [32, 26, 35, 36, 36, 28, 37, 32, 20, 37, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588084486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588101114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[1]R flush_numbers[0..12]: [48, 37, 52, 53, 54, 41, 55, 48, 24, 55, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588116561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[1]R generation[0..12]: [32, 26, 35, 36, 36, 28, 37, 32, 20, 37, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588131828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588148465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[2]R flush_numbers[0..12]: [48, 37, 52, 53, 54, 41, 55, 48, 24, 55, 49, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588164083Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[2]R generation[0..12]: [32, 26, 35, 36, 36, 28, 37, 32, 20, 37, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588179429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588195767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:03.588209323Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588230723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Next flush: 56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588244049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588257675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"":"downstairs","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588271912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588284527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"0","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588298484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"1","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588315561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"client":"2","":"downstairs","session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588330578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588344184Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"34a297ad-d848-4667-83e4-29a969a32aaa is now active with session: d1df33b7-e15e-438f-b63b-d6f0f0735ed9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588359211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588372107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354,"session_id":"d1df33b7-e15e-438f-b63b-d6f0f0735ed9"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588385803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} {"msg":"Activated sub_volume 34a297ad-d848-4667-83e4-29a969a32aaa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:03.588397289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 98 len 20480 data: 3 3 2 2 2 02/30 Write block 254 len 32768 data: 6 6 5 3 3 3 4 4 03/30 Read block 209 len 28672 04/30 Read block 524 len 32768 05/30 Read block 507 len 40960 06/30 Flush 07/30 Write block 386 len 8192 data: 4 4 08/30 Read block 457 len 4096 09/30 Read block 335 len 24576 10/30 Write block 126 len 20480 data: 4 5 5 5 5 11/30 Flush 12/30 Write block 151 len 16384 data: 4 3 3 5 13/30 Write block 59 len 20480 data: 5 5 4 4 4 14/30 Read block 513 len 8192 15/30 Write block 223 len 40960 data: 4 4 5 5 5 5 5 6 6 5 16/30 Read block 126 len 32768 17/30 Write block 24 len 36864 data: 4 4 5 4 4 3 3 4 4 18/30 Write block 277 len 28672 data: 5 4 4 3 3 4 4 19/30 Write block 223 len 20480 data: 5 5 6 6 6 20/30 Flush 21/30 Write block 73 len 32768 data: 4 4 5 4 6 6 6 6 22/30 Write block 90 len 12288 data: 8 8 8 23/30 Write block 91 len 16384 data: 9 9 8 7 24/30 Read block 97 len 36864 25/30 Read block 352 len 16384 26/30 Read block 148 len 40960 27/30 Write block 250 len 12288 data: 6 5 6 28/30 Write block 271 len 4096 data: 5 29/30 Write block 580 len 24576 data: 8 8 6 5 5 5 30/30 Read block 275 len 4096 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 3 30 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:03.600031984Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1354} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 26 38 38 37 58 58 F F F <--- 2 040-059 35 38 38 52 58 58 F F F <--- 3 060-079 36 38 38 53 58 58 F T T <--- 4 080-099 36 38 38 54 56 56 T T T <--- 5 100-119 28 38 38 41 56 56 T F F <--- 6 120-139 37 38 38 55 57 57 F F F <--- 7 140-159 32 38 38 48 58 58 F F F <--- 8 160-179 20 20 20 24 24 24 F F F 9 180-199 37 37 37 55 55 55 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 32 38 38 48 58 58 F F F <--- 12 240-259 37 38 38 55 56 56 T T T <--- 13 260-279 36 38 38 54 58 58 T T T <--- 14 280-299 36 38 38 53 58 58 F F F <--- 15 300-319 37 37 37 55 55 55 F F F 16 320-339 37 37 37 55 55 55 F F F 17 340-359 36 36 36 54 54 54 F F F 18 360-379 36 36 36 54 54 54 F F F 19 380-399 36 38 38 54 57 57 F F F <--- 20 400-419 30 30 30 44 44 44 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 35 35 35 52 52 52 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 32 32 32 48 48 48 F T T <--- Max gen: 38, Max flush: 58 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 38 38 38 58 58 58 F F F 2 040-059 38 38 38 58 58 58 F F F 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 38 38 38 57 57 57 F F F 7 140-159 38 38 38 58 58 58 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 37 37 37 55 55 55 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 38 38 38 58 58 58 F F F 12 240-259 39 39 39 59 59 59 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 38 38 38 58 58 58 F F F 15 300-319 37 37 37 55 55 55 F F F 16 320-339 37 37 37 55 55 55 F F F 17 340-359 36 36 36 54 54 54 F F F 18 360-379 36 36 36 54 54 54 F F F 19 380-399 38 38 38 57 57 57 F F F 20 400-419 30 30 30 44 44 44 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 26 26 37 37 37 F F F 24 480-499 32 32 32 48 48 48 F F F 25 500-519 32 32 32 48 48 48 F F F 26 520-539 35 35 35 52 52 52 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F F F 29 580-599 39 39 39 59 59 59 F F F Max gen: 39, Max flush: 59 Begin loop 20 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.936269957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:07.946893092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.946940423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957144478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957437811Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.95747833Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957490715Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957508503Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"Crucible d3b51c27-c51e-4fdc-9a2d-607f6dd318a2 has session id: 44d687af-e93b-4018-bdd0-dd98bcf007df","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957861284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Upstairs opts: Upstairs UUID: d3b51c27-c51e-4fdc-9a2d-607f6dd318a2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.95789643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Crucible stats registered with UUID: d3b51c27-c51e-4fdc-9a2d-607f6dd318a2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.957911987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:07.957925343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958079713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958114269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958129956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958230312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958244878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"d3b51c27-c51e-4fdc-9a2d-607f6dd318a2 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958265918Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958282215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958296892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958310648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958348756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958373597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958389024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"ds_connection connected from Ok(127.0.0.1:61459)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958404251Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"ds_connection connected from Ok(127.0.0.1:51536)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958419058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"ds_connection connected from Ok(127.0.0.1:38692)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958493842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.958989757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.95901792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959158404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959370581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959407778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[0]R flush_numbers[0..12]: [48, 58, 58, 59, 59, 56, 57, 58, 24, 55, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959528012Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[0]R generation[0..12]: [32, 38, 38, 39, 39, 38, 38, 38, 20, 37, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959547721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959564439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[1]R flush_numbers[0..12]: [48, 58, 58, 59, 59, 56, 57, 58, 24, 55, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959579796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[1]R generation[0..12]: [32, 38, 38, 39, 39, 38, 38, 38, 20, 37, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959595073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959611741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[2]R flush_numbers[0..12]: [48, 58, 58, 59, 59, 56, 57, 58, 24, 55, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959626988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[2]R generation[0..12]: [32, 38, 38, 39, 39, 38, 38, 38, 20, 37, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959642344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959658902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959672218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959692928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959706494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.95972014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"":"downstairs","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959734536Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959747272Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"0","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959761769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"1","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959779697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"client":"2","":"downstairs","session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959795294Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.95980877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"d3b51c27-c51e-4fdc-9a2d-607f6dd318a2 is now active with session: 44d687af-e93b-4018-bdd0-dd98bcf007df","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959823897Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959836763Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362,"session_id":"44d687af-e93b-4018-bdd0-dd98bcf007df"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959849969Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} {"msg":"Activated sub_volume d3b51c27-c51e-4fdc-9a2d-607f6dd318a2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:07.959861444Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 311 len 4096 data: 4 02/30 Write block 254 len 20480 data: 7 7 6 4 4 03/30 Read block 287 len 16384 04/30 Write block 38 len 32768 data: 5 5 5 4 5 3 4 3 05/30 Write block 473 len 36864 data: 3 3 3 3 4 4 5 5 6 06/30 Write block 531 len 36864 data: 4 5 5 3 4 4 4 4 5 07/30 Read block 24 len 20480 08/30 Write block 508 len 32768 data: 3 3 4 4 4 5 6 5 09/30 Read block 82 len 4096 10/30 Read block 182 len 8192 11/30 Write block 290 len 12288 data: 6 5 6 12/30 Flush 13/30 Flush 14/30 Write block 478 len 20480 data: 5 6 6 7 6 15/30 Read block 105 len 20480 16/30 Read block 486 len 36864 17/30 Write block 280 len 20480 data: 4 4 5 5 5 18/30 Read block 18 len 8192 19/30 Flush 20/30 Flush 21/30 Write block 409 len 20480 data: 4 4 4 4 3 22/30 Read block 474 len 40960 23/30 Write block 372 len 36864 data: 6 5 7 6 6 5 3 4 4 24/30 Write block 22 len 16384 data: 2 2 5 5 25/30 Write block 133 len 16384 data: 4 7 8 8 26/30 Read block 85 len 36864 27/30 Write block 187 len 24576 data: 2 2 5 3 3 3 28/30 Read block 96 len 20480 29/30 Read block 335 len 20480 30/30 Write block 571 len 4096 data: 6 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 2 29 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:07.971868559Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1362} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 38 40 40 58 60 60 F T T <--- 2 040-059 38 40 40 58 60 60 F F F <--- 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 38 38 38 57 57 57 F T T <--- 7 140-159 38 38 38 58 58 58 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 37 37 37 55 55 55 F T T <--- 10 200-219 33 33 33 49 49 49 F F F 11 220-239 38 38 38 58 58 58 F F F 12 240-259 39 40 40 59 60 60 T F F <--- 13 260-279 39 39 39 59 59 59 F F F 14 280-299 38 40 40 58 62 62 F F F <--- 15 300-319 37 40 40 55 60 60 T F F <--- 16 320-339 37 37 37 55 55 55 F F F 17 340-359 36 36 36 54 54 54 F F F 18 360-379 36 36 36 54 54 54 F T T <--- 19 380-399 38 38 38 57 57 57 F T T <--- 20 400-419 30 30 30 44 44 44 F T T <--- 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 26 40 40 37 62 62 F F F <--- 24 480-499 32 40 40 48 62 62 F F F <--- 25 500-519 32 40 40 48 60 60 F F F <--- 26 520-539 35 40 40 52 60 60 F F F <--- 27 540-559 33 33 33 49 49 49 F F F 28 560-579 28 28 28 41 41 41 F T T <--- 29 580-599 39 39 39 59 59 59 F F F Max gen: 40, Max flush: 62 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 40 40 60 60 60 F F F 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 41 41 63 63 63 F F F 7 140-159 38 38 38 58 58 58 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 41 41 41 63 63 63 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 38 38 38 58 58 58 F F F 12 240-259 40 40 40 60 60 60 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 62 62 62 F F F 15 300-319 40 40 40 60 60 60 F F F 16 320-339 37 37 37 55 55 55 F F F 17 340-359 36 36 36 54 54 54 F F F 18 360-379 41 41 41 63 63 63 F F F 19 380-399 41 41 41 63 63 63 F F F 20 400-419 41 41 41 63 63 63 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 35 35 52 52 52 F F F 23 460-479 40 40 40 62 62 62 F F F 24 480-499 40 40 40 62 62 62 F F F 25 500-519 40 40 40 60 60 60 F F F 26 520-539 40 40 40 60 60 60 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 41 41 41 63 63 63 F F F 29 580-599 39 39 39 59 59 59 F F F Max gen: 41, Max flush: 63 Begin loop 21 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.3149327Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:12.325576695Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.325623576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.335831332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336123835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336160912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336172928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336190466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"Crucible f819676c-a3b9-4644-bcef-bd1f39905436 has session id: 85cb57e4-9f29-461a-91f7-e2565d2b56a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336537023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Upstairs opts: Upstairs UUID: f819676c-a3b9-4644-bcef-bd1f39905436, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.33657435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Crucible stats registered with UUID: f819676c-a3b9-4644-bcef-bd1f39905436","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336590257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:12.336603403Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336754762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336784896Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336799993Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336915305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336930542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"f819676c-a3b9-4644-bcef-bd1f39905436 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336952172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336964968Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336978874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.336992711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337030808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.33705606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337071166Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43046)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337086583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:39119)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337104381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:38167)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337121859Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337819717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337857344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.337896622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.33819948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338235876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[0]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 58, 24, 63, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338280157Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[0]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 38, 20, 41, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338297525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338314472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[1]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 58, 24, 63, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338329949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[1]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 38, 20, 41, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338345746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338361934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[2]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 58, 24, 63, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338377401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[2]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 38, 20, 41, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338395729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338412257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {Wait for a query_work_queue command to finish before sending IO "msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338431425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338445742Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Next flush: 64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338459238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338472434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"":"downstairs","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338486471Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338499046Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"0","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338513523Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"1","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338527219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"client":"2","":"downstairs","session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338543296Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338556863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"f819676c-a3b9-4644-bcef-bd1f39905436 is now active with session: 85cb57e4-9f29-461a-91f7-e2565d2b56a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338571609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338584395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370,"session_id":"85cb57e4-9f29-461a-91f7-e2565d2b56a9"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338597881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} {"msg":"Activated sub_volume f819676c-a3b9-4644-bcef-bd1f39905436","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:12.338611057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 143 len 40960 data: 4 4 3 3 3 3 3 4 5 4 02/30 Read block 333 len 36864 03/30 Write block 497 len 20480 data: 7 8 8 8 5 04/30 Read block 5 len 36864 05/30 Read block 79 len 32768 06/30 Read block 259 len 4096 07/30 Read block 314 len 8192 08/30 Write block 473 len 12288 data: 4 4 4 09/30 Read block 364 len 12288 10/30 Read block 415 len 32768 11/30 Read block 120 len 28672 12/30 Write block 303 len 12288 data: 3 2 2 13/30 Read block 167 len 8192 14/30 Read block 392 len 28672 15/30 Flush 16/30 Write block 593 len 24576 data: 4 5 5 5 4 3 17/30 Write block 387 len 4096 data: 5 18/30 Write block 566 len 20480 data: 4 4 6 8 7 19/30 Write block 452 len 4096 data: 4 20/30 Read block 167 len 36864 21/30 Read block 497 len 8192 22/30 Read block 78 len 20480 23/30 Write block 244 len 8192 data: 3 3 24/30 Write block 335 len 28672 data: 2 2 2 4 5 5 5 25/30 Flush 26/30 Write block 464 len 40960 data: 4 4 4 4 4 4 5 5 4 5 27/30 Write block 569 len 12288 data: 9 8 7 28/30 Read block 183 len 16384 29/30 Write block 186 len 12288 data: 3 3 3 30/30 Write block 379 len 24576 data: 5 5 3 4 4 4 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 2 28 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:12.350515595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1370} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 40 40 60 60 60 F F F 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 41 41 63 63 63 F F F 7 140-159 38 42 42 58 64 64 T F F <--- 8 160-179 20 20 20 24 24 24 F F F 9 180-199 41 41 41 63 63 63 F T T <--- 10 200-219 33 33 33 49 49 49 F F F 11 220-239 38 38 38 58 58 58 F F F 12 240-259 40 42 42 60 65 65 F F F <--- 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 62 62 62 F F F 15 300-319 40 42 42 60 64 64 F F F <--- 16 320-339 37 42 42 55 65 65 F F F <--- 17 340-359 36 42 42 54 65 65 F F F <--- 18 360-379 41 41 41 63 63 63 F T T <--- 19 380-399 41 42 42 63 65 65 F T T <--- 20 400-419 41 41 41 63 63 63 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 35 42 42 52 65 65 F F F <--- 23 460-479 40 42 42 62 64 64 F T T <--- 24 480-499 40 42 42 62 64 64 F F F <--- 25 500-519 40 42 42 60 64 64 F F F <--- 26 520-539 40 40 40 60 60 60 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 41 42 42 63 65 65 F T T <--- 29 580-599 39 42 42 59 65 65 F F F <--- Max gen: 42, Max flush: 65 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 48 48 48 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 40 40 60 60 60 F F F 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 41 41 63 63 63 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 20 20 20 24 24 24 F F F 9 180-199 43 43 43 66 66 66 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 38 38 38 58 58 58 F F F 12 240-259 42 42 42 65 65 65 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 62 62 62 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 42 42 42 65 65 65 F F F 17 340-359 42 42 42 65 65 65 F F F 18 360-379 43 43 43 66 66 66 F F F 19 380-399 43 43 43 66 66 66 F F F 20 400-419 41 41 41 63 63 63 F F F 21 420-439 35 35 35 52 52 52 F F F 22 440-459 42 42 42 65 65 65 F F F 23 460-479 43 43 43 66 66 66 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 42 42 42 64 64 64 F F F 26 520-539 40 40 40 60 60 60 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 43, Max flush: 66 Begin loop 22 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.689492187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.701359328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.701682285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.701724124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70173737Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.701755428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"Crucible 93ab757b-c3d5-4846-b821-58da7198f5e5 has session id: 52a4a75f-ab81-4fb5-80fd-d632e44b1d32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702122596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Upstairs opts: Upstairs UUID: 93ab757b-c3d5-4846-b821-58da7198f5e5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702156721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Crucible stats registered with UUID: 93ab757b-c3d5-4846-b821-58da7198f5e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702173749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:16.702187355Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702356732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702388437Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702404274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702495965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702512913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"93ab757b-c3d5-4846-b821-58da7198f5e5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702530901Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702543527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702557343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70257129Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702585336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702617961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702642042Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"ds_connection connected from Ok(127.0.0.1:48992)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702658009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"ds_connection connected from Ok(127.0.0.1:40135)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702673306Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"ds_connection connected from Ok(127.0.0.1:41986)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.702789909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703380028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70340727Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703424088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703786393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703803411Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[0]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 64, 24, 66, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703832544Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[0]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 42, 20, 43, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703848161Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703864538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[1]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 64, 24, 66, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703879835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[1]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 42, 20, 43, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703895002Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70391126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[2]R flush_numbers[0..12]: [48, 63, 60, 59, 59, 56, 63, 64, 24, 66, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703926657Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[2]R generation[0..12]: [32, 41, 40, 39, 39, 38, 41, 42, 20, 43, 33, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703942024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703958311Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.703971647Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:16.703985854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Next flush: 67","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704023161Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704036817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"":"downstairs","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704051233Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70406593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"0","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704079926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"1","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704093853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"client":"2","":"downstairs","session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"All required reconciliation work is completed","v":0Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70411081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704124337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"93ab757b-c3d5-4846-b821-58da7198f5e5 is now active with session: 52a4a75f-ab81-4fb5-80fd-d632e44b1d32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704139073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704151579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378,"session_id":"52a4a75f-ab81-4fb5-80fd-d632e44b1d32"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.704164905Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} {"msg":"Activated sub_volume 93ab757b-c3d5-4846-b821-58da7198f5e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:16.70417605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 428 len 16384 data: 5 5 5 5 02/30 Write block 189 len 12288 data: 6 4 4 03/30 Write block 411 len 36864 data: 5 5 4 4 4 3 2 2 2 04/30 Write block 220 len 20480 data: 4 4 4 6 6 05/30 Write block 75 len 24576 data: 6 5 7 7 7 7 06/30 Write block 246 len 12288 data: 5 5 5 07/30 Write block 229 len 24576 data: 6 7 7 6 4 5 08/30 Write block 370 len 20480 data: 6 6 7 6 8 09/30 Write block 3 len 28672 data: 2 2 2 3 2 4 5 10/30 Write block 327 len 24576 data: 6 6 6 5 3 3 11/30 Read block 198 len 40960 12/30 Write block 3 len 40960 data: 3 3 3 4 3 5 6 5 5 6 13/30 Read block 523 len 40960 14/30 Read block 221 len 20480 15/30 Write block 225 len 28672 data: 7 7 7 6 7 8 8 16/30 Flush 17/30 Write block 287 len 20480 data: 7 7 8 7 6 18/30 Read block 160 len 24576 19/30 Write block 396 len 28672 data: 3 3 3 3 3 3 2 20/30 Write block 415 len 24576 data: 5 4 3 3 3 2 21/30 Read block 523 len 32768 22/30 Read block 404 len 20480 23/30 Write block 83 len 12288 data: 4 5 6 24/30 Write block 407 len 28672 data: 5 5 5 5 6 6 5 25/30 Read block 423 len 32768 26/30 Write block 164 len 12288 data: 3 3 3 27/30 Write block 470 len 16384 data: 6 6 5 6 28/30 Write block 247 len 40960 data: 6 6 6 7 6 7 5 8 8 7 29/30 Read block 514 len 28672 30/30 Write block 358 len 32768 data: 4 3 4 3 3 2 4 3 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Write 5 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 29 29 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:16.715714112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1378} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 32 67 67 48 F F F <--- 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 40 40 60 60 60 F F F 3 060-079 44 44 39 67 67 59 F F F <--- 4 080-099 44 44 39 67 67 59 T T F <--- 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 41 41 63 63 63 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 20 20 20 24 24 24 T T F <--- 9 180-199 44 44 43 67 67 66 F F T <--- 10 200-219 33 33 33 49 49 49 F F F 11 220-239 44 44 38 67 67 58 F F F <--- 12 240-259 44 44 42 67 67 65 T T F <--- 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 62 62 62 T T F <--- 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 42 67 67 65 F F F <--- 17 340-359 42 42 42 65 65 65 T T F <--- 18 360-379 44 44 43 67 67 66 T T F <--- 19 380-399 43 43 43 66 66 66 T T F <--- 20 400-419 44 44 41 67 67 63 T T T <--- 21 420-439 44 44 35 67 67 52 T T T <--- 22 440-459 42 42 42 65 65 65 F F F 23 460-479 43 43 43 66 66 66 T T F <--- 24 480-499 42 42 42 64 64 64 F F F 25 500-519 42 42 42 64 64 64 F F F 26 520-539 40 40 40 60 60 60 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 44, Max flush: 67 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 67 67 67 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 40 40 60 60 60 F F F 3 060-079 44 44 44 67 67 67 F F F 4 080-099 45 45 45 68 68 68 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 41 41 63 63 63 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 45 45 45 68 68 68 F F F 9 180-199 44 44 44 67 67 67 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 45 45 45 68 68 68 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 44 67 67 67 F F F 17 340-359 45 45 45 68 68 68 F F F 18 360-379 45 45 45 68 68 68 F F F 19 380-399 45 45 45 68 68 68 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 45 45 45 68 68 68 F F F 22 440-459 42 42 42 65 65 65 F F F 23 460-479 45 45 45 68 68 68 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 42 42 42 64 64 64 F F F 26 520-539 40 40 40 60 60 60 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 45, Max flush: 68 Begin loop 23 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.052102714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:21.062754612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.062801793Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073025647Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073320731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073360239Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073372475Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073392944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"Crucible 0e966a8d-0a84-4cd4-870b-c89f05c24652 has session id: 91d0bfe1-8319-445e-975a-936caa3be5e3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073750997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Upstairs opts: Upstairs UUID: 0e966a8d-0a84-4cd4-870b-c89f05c24652, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073786813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Crucible stats registered with UUID: 0e966a8d-0a84-4cd4-870b-c89f05c24652","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.07380226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:21.073816056Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073950618Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073982292Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.073997269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.07413174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074147047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"0e966a8d-0a84-4cd4-870b-c89f05c24652 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074168247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074181013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074195029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074208855Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074249834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074294804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074310181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:53332)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074324978Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61536)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074339645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:36613)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.074354341Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075023736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075052029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075068757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075377297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075430251Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[0]R flush_numbers[0..12]: [67, 63, 60, 67, 68, 56, 63, 64, 68, 67, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075482535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[0]R generation[0..12]: [44, 41, 40, 44, 45, 38, 41, 42, 45, 44, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075501233Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075518381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[1]R flush_numbers[0..12]: [67, 63, 60, 67, 68, 56, 63, 64, 68, 67, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075533918Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[1]R generation[0..12]: [44, 41, 40, 44, 45, 38, 41, 42, 45, 44, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075554418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075570985Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[2]R flush_numbers[0..12]: [67, 63, 60, 67, 68, 56, 63, 64, 68, 67, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075586182Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[2]R generation[0..12]: [44, 41, 40, 44, 45, 38, 41, 42, 45, 44, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075601449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075617806Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075631353Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075650801Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Next flush: 69","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075664458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075677614Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"":"downstairs","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.07569153Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075704636Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"0","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075718882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"1","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075732809Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"client":"2","":"downstairs","session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075749947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075763363Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"0e966a8d-0a84-4cd4-870b-c89f05c24652 is now active with session: 91d0bfe1-8319-445e-975a-936caa3be5e3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.07577985Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075792566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386,"session_id":"91d0bfe1-8319-445e-975a-936caa3be5e3"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075805642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} {"msg":"Activated sub_volume 0e966a8d-0a84-4cd4-870b-c89f05c24652","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:21.075817037Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 461 len 16384 data: 2 2 3 5 02/30 Read block 488 len 20480 03/30 Flush 04/30 Write block 95 len 8192 data: 5 3 05/30 Flush 06/30 Write block 130 len 36864 data: 6 6 4 5 8 9 9 6 6 07/30 Flush 08/30 Flush 09/30 Write block 362 len 20480 data: 4 3 5 4 3 10/30 Write block 55 len 24576 data: 6 5 4 5 6 6 11/30 Write block 170 len 32768 data: 3 3 4 4 4 3 4 4 12/30 Read block 546 len 32768 13/30 Write block 503 len 40960 data: 5 5 6 4 5 4 4 5 5 5 14/30 Write block 426 len 28672 data: 6 6 6 6 6 6 5 15/30 Read block 364 len 12288 16/30 Flush 17/30 Write block 469 len 16384 data: 5 7 7 6 18/30 Flush 19/30 Write block 458 len 40960 data: 4 3 2 3 3 4 6 5 5 5 20/30 Read block 515 len 24576 21/30 Write block 382 len 36864 data: 5 5 5 3 5 6 5 5 4 22/30 Write block 180 len 28672 data: 5 5 5 5 4 4 4 23/30 Read block 222 len 12288 24/30 Read block 109 len 28672 25/30 Flush 26/30 Write block 479 len 16384 data: 7 7 8 7 27/30 Read block 69 len 36864 28/30 Write block 425 len 36864 data: 7 7 7 7 7 7 7 6 5 29/30 Write block 169 len 16384 data: 3 4 4 5 30/30 Write block 524 len 16384 data: 3 3 2 2 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:21.08856713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1386} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 67 67 67 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 40 46 46 60 73 73 F F F <--- 3 060-079 44 46 46 67 73 73 F F F <--- 4 080-099 45 46 46 68 70 70 F F F <--- 5 100-119 38 38 38 56 56 56 F F F 6 120-139 41 46 46 63 71 71 F F F <--- 7 140-159 42 42 42 64 64 64 F F F 8 160-179 45 46 46 68 73 73 F T T <--- 9 180-199 44 46 46 67 75 75 F F F <--- 10 200-219 33 33 33 49 49 49 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 45 45 45 68 68 68 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 44 67 67 67 F F F 17 340-359 45 45 45 68 68 68 F F F 18 360-379 45 46 46 68 73 73 F F F <--- 19 380-399 45 46 46 68 75 75 F F F <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 45 46 46 68 73 73 F T T <--- 22 440-459 42 46 46 65 75 75 F F F <--- 23 460-479 45 46 46 68 75 75 F T T <--- 24 480-499 42 42 42 64 64 64 F T T <--- 25 500-519 42 46 46 64 73 73 F F F <--- 26 520-539 40 40 40 60 60 60 F T T <--- 27 540-559 33 33 33 49 49 49 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 46, Max flush: 75 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 67 67 67 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 46 46 46 73 73 73 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 F F F 9 180-199 46 46 46 75 75 75 F F F 10 200-219 33 33 33 49 49 49 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 45 45 45 68 68 68 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 44 67 67 67 F F F 17 340-359 45 45 45 68 68 68 F F F 18 360-379 46 46 46 73 73 73 F F F 19 380-399 46 46 46 75 75 75 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 46 46 46 75 75 75 F F F 23 460-479 47 47 47 76 76 76 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 46 46 46 73 73 73 F F F 26 520-539 47 47 47 76 76 76 F F F 27 540-559 33 33 33 49 49 49 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 47, Max flush: 76 Begin loop 24 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.426836429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.438695677Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439005588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439046977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439059212Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439077011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"Crucible 83326dac-e3dc-410d-8b82-adf52d2f1267 has session id: faa308a4-0676-49fc-b42a-bbaf7d4afa20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439446259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Upstairs opts: Upstairs UUID: 83326dac-e3dc-410d-8b82-adf52d2f1267, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439473991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Crucible stats registered with UUID: 83326dac-e3dc-410d-8b82-adf52d2f1267","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439489058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:25.439501804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439667729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439702015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439720854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439815026Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439829223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"83326dac-e3dc-410d-8b82-adf52d2f1267 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439850553Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439863438Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439877465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439891481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439905167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439934951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439949308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"ds_connection connected from Ok(127.0.0.1:45441)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439964184Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"ds_connection connected from Ok(127.0.0.1:37075)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439979391Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"ds_connection connected from Ok(127.0.0.1:45703)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.439994418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.440731023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.440755154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.440774963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441150244Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441166371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[0]R flush_numbers[0..12]: [67, 63, 73, 73, 70, 56, 71, 64, 76, 75, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441191743Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[0]R generation[0..12]: [44, 41, 46, 46, 46, 38, 46, 42, 47, 46, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.44120745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441224428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[1]R flush_numbers[0..12]: [67, 63, 73, 73, 70, 56, 71, 64, 76, 75, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441239905Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[1]R generation[0..12]: [44, 41, 46, 46, 46, 38, 46, 42, 47, 46, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441256052Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.44127257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[2]R flush_numbers[0..12]: [67, 63, 73, 73, 70, 56, 71, 64, 76, 75, 49, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441287957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[2]R generation[0..12]: [44, 41, 46, 46, 46, 38, 46, 42, 47, 46, 33, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441303294Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441319581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441333207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441347074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 77","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441368483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.44138248Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"":"downstairs","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441397217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441409812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"0","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441424159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"1","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441438325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"client":"2","":"downstairs","session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441452492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441465748Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"83326dac-e3dc-410d-8b82-adf52d2f1267 is now active with session: faa308a4-0676-49fc-b42a-bbaf7d4afa20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441483726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441496612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394,"session_id":"faa308a4-0676-49fc-b42a-bbaf7d4afa20"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441509698Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} {"msg":"Activated sub_volume 83326dac-e3dc-410d-8b82-adf52d2f1267","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:25.441521243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 503 len 24576 data: 6 6 7 5 6 5 02/30 Read block 62 len 40960 03/30 Flush 04/30 Write block 507 len 32768 data: 7 6 5 6 6 6 6 7 05/30 Read block 340 len 8192 06/30 Read block 459 len 20480 07/30 Write block 448 len 24576 data: 4 4 4 4 5 5 08/30 Read block 212 len 20480 09/30 Write block 210 len 4096 data: 5 10/30 Read block 421 len 32768 11/30 Write block 345 len 8192 data: 3 3 12/30 Flush 13/30 Write block 542 len 16384 data: 5 7 7 7 14/30 Write block 231 len 36864 data: 9 7 5 6 5 5 4 4 2 15/30 Read block 479 len 8192 16/30 Flush 17/30 Write block 53 len 12288 data: 6 7 7 18/30 Read block 521 len 40960 19/30 Read block 2 len 28672 20/30 Read block 553 len 12288 21/30 Read block 368 len 12288 22/30 Read block 491 len 20480 23/30 Read block 299 len 36864 24/30 Read block 483 len 40960 25/30 Flush 26/30 Write block 461 len 4096 data: 4 27/30 Write block 275 len 8192 data: 5 5 28/30 Read block 591 len 36864 29/30 Read block 175 len 16384 30/30 Read block 384 len 12288 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Read 10 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:25.453410134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1394} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 67 67 67 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 46 80 80 73 F F F <--- 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 F F F 9 180-199 46 46 46 75 75 75 F F F 10 200-219 48 48 33 78 78 49 F F F <--- 11 220-239 48 48 44 79 79 67 F F F <--- 12 240-259 45 45 45 68 68 68 F F F 13 260-279 39 39 39 59 59 59 T T F <--- 14 280-299 45 45 45 68 68 68 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 44 67 67 67 F F F 17 340-359 48 48 45 78 78 68 F F F <--- 18 360-379 46 46 46 73 73 73 F F F 19 380-399 46 46 46 75 75 75 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 48 48 46 78 78 75 F F F <--- 23 460-479 47 47 47 76 76 76 T T F <--- 24 480-499 47 47 47 76 76 76 F F F 25 500-519 48 48 46 78 78 73 F F F <--- 26 520-539 47 47 47 76 76 76 F F F 27 540-559 48 48 33 79 79 49 F F F <--- 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 48, Max flush: 80 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 67 67 67 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 48 80 80 80 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 F F F 9 180-199 46 46 46 75 75 75 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 48 48 48 79 79 79 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 49 49 49 81 81 81 F F F 14 280-299 45 45 45 68 68 68 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 44 44 44 67 67 67 F F F 17 340-359 48 48 48 78 78 78 F F F 18 360-379 46 46 46 73 73 73 F F F 19 380-399 46 46 46 75 75 75 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 48 48 48 78 78 78 F F F 23 460-479 49 49 49 81 81 81 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 48 48 48 78 78 78 F F F 26 520-539 47 47 47 76 76 76 F F F 27 540-559 48 48 48 79 79 79 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 42 42 42 65 65 65 F F F Max gen: 49, Max flush: 81 Begin loop 25 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.769242341Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.780962595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781299779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781343989Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781356085Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781373943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"Crucible e78b8143-20b0-4404-bad2-1a83baf55a40 has session id: b5be62c9-c4e9-4285-baab-6cd825f0efa8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781713417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Upstairs opts: Upstairs UUID: e78b8143-20b0-4404-bad2-1a83baf55a40, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781748903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Crucible stats registered with UUID: e78b8143-20b0-4404-bad2-1a83baf55a40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.78176466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:29.781778107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.78194012Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781975667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.781991194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782102874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782119522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"e78b8143-20b0-4404-bad2-1a83baf55a40 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782150276Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782164432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782178469Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782192285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782205761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782219908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782236685Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"ds_connection connected from Ok(127.0.0.1:63851)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782292971Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"ds_connection connected from Ok(127.0.0.1:58808)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782318873Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"ds_connection connected from Ok(127.0.0.1:45752)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.782397398Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783022573Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783050406Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783066813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783379936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783395103Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[0]R flush_numbers[0..12]: [67, 63, 80, 73, 70, 56, 71, 64, 76, 75, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783421274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[0]R generation[0..12]: [44, 41, 48, 46, 46, 38, 46, 42, 47, 46, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783436802Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783453059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[1]R flush_numbers[0..12]: [67, 63, 80, 73, 70, 56, 71, 64, 76, 75, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783468546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[1]R generation[0..12]: [44, 41, 48, 46, 46, 38, 46, 42, 47, 46, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783483763Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783503492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[2]R flush_numbers[0..12]: [67, 63, 80, 73, 70, 56, 71, 64, 76, 75, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783518999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[2]R generation[0..12]: [44, 41, 48, 46, 46, 38, 46, 42, 47, 46, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783534346Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783550443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783564089Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783577766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Next flush: 82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783590962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:29.783604348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"":"downstairs","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783623516Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783636212Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"0","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783650119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"1","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783664035Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"client":"2","":"downstairs","session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783678201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783691748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"e78b8143-20b0-4404-bad2-1a83baf55a40 is now active with session: b5be62c9-c4e9-4285-baab-6cd825f0efa8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783706414Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783723372Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402,"session_id":"b5be62c9-c4e9-4285-baab-6cd825f0efa8"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783736998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} {"msg":"Activated sub_volume e78b8143-20b0-4404-bad2-1a83baf55a40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:29.783748293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 276 len 28672 data: 6 6 5 5 5 5 6 02/30 Read block 590 len 12288 03/30 Write block 191 len 4096 data: 5 04/30 Read block 348 len 32768 05/30 Read block 276 len 40960 06/30 Flush 07/30 Write block 390 len 20480 data: 5 4 4 3 4 08/30 Write block 5 len 16384 data: 4 5 4 6 09/30 Read block 471 len 40960 10/30 Write block 581 len 24576 data: 9 7 6 6 6 6 11/30 Read block 155 len 8192 12/30 Read block 452 len 28672 13/30 Read block 196 len 24576 14/30 Write block 337 len 16384 data: 3 5 6 6 15/30 Read block 474 len 36864 16/30 Read block 328 len 8192 17/30 Read block 268 len 40960 18/30 Read block 312 len 40960 19/30 Read block 283 len 24576 20/30 Read block 69 len 8192 21/30 Write block 529 len 28672 data: 3 3 5 6 6 4 5 22/30 Read block 494 len 8192 23/30 Flush 24/30 Write block 517 len 4096 data: 4 25/30 Read block 571 len 28672 26/30 Write block 320 len 40960 data: 8 7 7 6 6 6 8 7 7 7 27/30 Read block 587 len 16384 28/30 Read block 72 len 24576 29/30 Write block 378 len 32768 data: 4 6 6 4 6 6 6 4 30/30 Read block 239 len 12288 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 30 3 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:29.79659405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1402} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 44 50 83 67 83 F F F <--- 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 48 80 80 80 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 F F F 9 180-199 50 46 50 82 75 82 F T F <--- 10 200-219 48 48 48 78 78 78 F F F 11 220-239 48 48 48 79 79 79 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 50 49 50 82 81 82 F T F <--- 14 280-299 50 45 50 82 68 82 F T F <--- 15 300-319 42 42 42 64 64 64 F F F 16 320-339 50 44 50 83 67 83 T F T <--- 17 340-359 50 48 50 83 78 83 F F F <--- 18 360-379 46 46 46 73 73 73 T F T <--- 19 380-399 50 46 50 83 75 83 T F T <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 48 48 48 78 78 78 F F F 23 460-479 49 49 49 81 81 81 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 48 48 48 78 78 78 T F T <--- 26 520-539 50 47 50 83 76 83 F F F <--- 27 540-559 48 48 48 79 79 79 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 42 50 83 65 83 F F F <--- Max gen: 50, Max flush: 83 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 83 83 83 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 48 80 80 80 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 38 38 38 56 56 56 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 F F F 9 180-199 50 50 50 82 82 82 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 48 48 48 79 79 79 F F F 12 240-259 45 45 45 68 68 68 F F F 13 260-279 50 50 50 82 82 82 F F F 14 280-299 50 50 50 82 82 82 F F F 15 300-319 42 42 42 64 64 64 F F F 16 320-339 51 51 51 84 84 84 F F F 17 340-359 50 50 50 83 83 83 F F F 18 360-379 51 51 51 84 84 84 F F F 19 380-399 51 51 51 84 84 84 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 48 48 48 78 78 78 F F F 23 460-479 49 49 49 81 81 81 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 50 50 50 83 83 83 F F F 27 540-559 48 48 48 79 79 79 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 50 50 83 83 83 F F F Max gen: 51, Max flush: 84 Begin loop 26 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.117607345Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129257957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129601744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129640271Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129652887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129670505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"Crucible 0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8 has session id: 65bdff0b-47d2-4419-a779-222041e132bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.129987109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Upstairs opts: Upstairs UUID: 0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130014662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Crucible stats registered with UUID: 0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130029619Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:34.130042665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130189371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130212252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130226978Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.13034043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130355817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130376817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130389622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130403339Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130417245Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130431101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130467678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130490528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:55549)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130506135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:44488)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130612534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:62784)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.130667009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"io task","client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131274235Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131310622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131327429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131633268Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131675768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[0]R flush_numbers[0..12]: [83, 63, 80, 73, 70, 56, 71, 64, 76, 82, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.13174581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[0]R generation[0..12]: [50, 41, 48, 46, 46, 38, 46, 42, 47, 50, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131763198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131779895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[1]R flush_numbers[0..12]: [83, 63, 80, 73, 70, 56, 71, 64, 76, 82, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131795342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[1]R generation[0..12]: [50, 41, 48, 46, 46, 38, 46, 42, 47, 50, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131816242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131832609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[2]R flush_numbers[0..12]: [83, 63, 80, 73, 70, 56, 71, 64, 76, 82, 78, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131847746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[2]R generation[0..12]: [50, 41, 48, 46, 46, 38, 46, 42, 47, 50, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131862823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.13187914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131892476Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131912836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Next flush: 85","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131926172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131939488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"":"downstairs","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131955435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131968351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"0","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131982557Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"1","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.131996534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"client":"2","":"downstairs","session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.132013001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.132026597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8 is now active with session: 65bdff0b-47d2-4419-a779-222041e132bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.132041654Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.13205429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410,"session_id":"65bdff0b-47d2-4419-a779-222041e132bd"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.132067716Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} {"msg":"Activated sub_volume 0f8d12ad-6522-4fa5-8ee8-e00ef4c392e8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:34.132079231Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 310 len 24576 data: 4 5 4 3 4 5 02/30 Read block 552 len 32768 03/30 Read block 208 len 20480 04/30 Write block 251 len 28672 data: 7 8 6 9 9 8 5 05/30 Read block 322 len 36864 06/30 Flush 07/30 Write block 262 len 32768 data: 3 4 4 4 5 5 5 5 08/30 Flush 09/30 Write block 245 len 24576 data: 4 6 7 7 7 8 10/30 Read block 448 len 12288 11/30 Read block 194 len 32768 12/30 Read block 540 len 20480 13/30 Flush 14/30 Write block 246 len 32768 data: 7 8 8 8 9 8 9 7 15/30 Write block 221 len 28672 data: 5 5 7 7 8 8 8 16/30 Read block 278 len 16384 17/30 Write block 383 len 24576 data: 7 7 5 6 7 6 18/30 Read block 540 len 4096 19/30 Write block 96 len 24576 data: 4 3 4 4 3 3 20/30 Write block 358 len 4096 data: 5 21/30 Read block 213 len 36864 22/30 Read block 447 len 4096 23/30 Read block 563 len 8192 24/30 Write block 443 len 40960 data: 4 4 3 2 2 5 5 5 5 6 25/30 Read block 507 len 8192 26/30 Read block 571 len 40960 27/30 Write block 346 len 16384 data: 4 3 5 5 28/30 Read block 14 len 16384 29/30 Write block 165 len 28672 data: 4 4 4 4 4 5 5 30/30 Read block 157 len 28672 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Write 7 Done Sent Sent false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 2 29 Done 30 3 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:34.145489825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1410} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 83 83 83 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 48 80 80 80 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 46 46 46 70 70 70 T F T <--- 5 100-119 38 38 38 56 56 56 T F T <--- 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 47 47 47 76 76 76 T F T <--- 9 180-199 50 50 50 82 82 82 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 48 48 48 79 79 79 T F T <--- 12 240-259 52 45 52 87 68 87 T F T <--- 13 260-279 52 50 52 86 82 86 F F F <--- 14 280-299 50 50 50 82 82 82 F F F 15 300-319 52 42 52 85 64 85 F T F <--- 16 320-339 51 51 51 84 84 84 F F F 17 340-359 50 50 50 83 83 83 T F T <--- 18 360-379 51 51 51 84 84 84 F F F 19 380-399 51 51 51 84 84 84 T F T <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 48 48 48 78 78 78 T F T <--- 23 460-479 49 49 49 81 81 81 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 50 50 50 83 83 83 F F F 27 540-559 48 48 48 79 79 79 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 50 50 83 83 83 F F F Max gen: 52, Max flush: 87 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 83 83 83 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 48 48 48 80 80 80 F F F 3 060-079 46 46 46 73 73 73 F F F 4 080-099 53 53 53 88 88 88 F F F 5 100-119 53 53 53 88 88 88 F F F 6 120-139 46 46 46 71 71 71 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 53 53 53 88 88 88 F F F 9 180-199 50 50 50 82 82 82 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 53 53 53 88 88 88 F F F 12 240-259 53 53 53 88 88 88 F F F 13 260-279 52 52 52 86 86 86 F F F 14 280-299 50 50 50 82 82 82 F F F 15 300-319 52 52 52 85 85 85 F F F 16 320-339 51 51 51 84 84 84 F F F 17 340-359 53 53 53 88 88 88 F F F 18 360-379 51 51 51 84 84 84 F F F 19 380-399 53 53 53 88 88 88 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 49 49 49 81 81 81 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 50 50 50 83 83 83 F F F 27 540-559 48 48 48 79 79 79 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 50 50 83 83 83 F F F Max gen: 53, Max flush: 88 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.466323348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.477985466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478304671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478341748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478354074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478379566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"Crucible 574c7fe0-5158-4b26-9ffd-e78b762d815f has session id: 6621f062-577b-4219-b811-f4e74b233323","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478714738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Upstairs opts: Upstairs UUID: 574c7fe0-5158-4b26-9ffd-e78b762d815f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478752375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Crucible stats registered with UUID: 574c7fe0-5158-4b26-9ffd-e78b762d815f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478768152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:38.478781659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478943632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.478978428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.47900418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479097232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479111939Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"574c7fe0-5158-4b26-9ffd-e78b762d815f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479133429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479146395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479160321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479174297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479188003Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.47920267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479217627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"ds_connection connected from Ok(127.0.0.1:63073)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479261157Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"ds_connection connected from Ok(127.0.0.1:46587)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.47940093Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"ds_connection connected from Ok(127.0.0.1:58242)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479428523Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"io task","client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.479991019Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480032098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480141968Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480333105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480371112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[0]R flush_numbers[0..12]: [83, 63, 80, 73, 88, 88, 71, 64, 88, 82, 78, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480535497Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[0]R generation[0..12]: [50, 41, 48, 46, 53, 53, 46, 42, 53, 50, 48, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480552284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480569342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[1]R flush_numbers[0..12]: [83, 63, 80, 73, 88, 88, 71, 64, 88, 82, 78, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480585209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[1]R generation[0..12]: [50, 41, 48, 46, 53, 53, 46, 42, 53, 50, 48, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480600296Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480616474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[2]R flush_numbers[0..12]: [83, 63, 80, 73, 88, 88, 71, 64, 88, 82, 78, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480634962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[2]R generation[0..12]: [50, 41, 48, 46, 53, 53, 46, 42, 53, 50, 48, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480650389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480666596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480680003Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480699932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Next flush: 89","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480712998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480726804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"":"downstairs","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.48074067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480753696Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"0","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480767802Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"1","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480781549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"client":"2","":"downstairs","session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:02:38.480795515Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480811642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"574c7fe0-5158-4b26-9ffd-e78b762d815f is now active with session: 6621f062-577b-4219-b811-f4e74b233323","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480826529Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480839295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418,"session_id":"6621f062-577b-4219-b811-f4e74b233323"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480854472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} {"msg":"Activated sub_volume 574c7fe0-5158-4b26-9ffd-e78b762d815f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:38.480866107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 373 len 32768 data: 7 9 7 7 6 5 7 7 02/30 Write block 273 len 28672 data: 6 6 6 7 7 6 6 03/30 Read block 318 len 16384 04/30 Write block 59 len 28672 data: 7 7 5 5 5 6 6 05/30 Read block 80 len 28672 06/30 Read block 159 len 4096 07/30 Read block 354 len 16384 08/30 Write block 309 len 16384 data: 4 5 6 5 09/30 Write block 125 len 12288 data: 4 5 6 10/30 Read block 298 len 24576 11/30 Read block 286 len 12288 12/30 Write block 160 len 16384 data: 3 2 2 3 13/30 Write block 271 len 36864 data: 6 6 7 7 7 8 8 7 7 14/30 Read block 313 len 4096 15/30 Write block 163 len 16384 data: 4 4 5 5 16/30 Write block 463 len 12288 data: 5 7 6 17/30 Read block 245 len 8192 18/30 Read block 381 len 20480 19/30 Flush 20/30 Write block 522 len 4096 data: 4 21/30 Read block 282 len 24576 22/30 Write block 534 len 36864 data: 5 6 5 5 5 6 5 5 6 23/30 Flush 24/30 Flush 25/30 Write block 227 len 4096 data: 9 26/30 Write block 113 len 36864 data: 4 4 4 4 4 3 3 3 4 27/30 Read block 191 len 20480 28/30 Read block 283 len 16384 29/30 Read block 574 len 4096 30/30 Read block 590 len 12288 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 7 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:38.492748745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1418} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 83 83 83 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 54 54 48 89 89 80 F F F <--- 3 060-079 54 54 46 89 89 73 F F F <--- 4 080-099 53 53 53 88 88 88 F F F 5 100-119 53 53 53 88 88 88 T T F <--- 6 120-139 54 54 46 89 89 71 T T F <--- 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 53 89 89 88 F F F <--- 9 180-199 50 50 50 82 82 82 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 53 53 53 88 88 88 T T F <--- 12 240-259 53 53 53 88 88 88 F F F 13 260-279 54 54 52 89 89 86 F F F <--- 14 280-299 50 50 50 82 82 82 F F F 15 300-319 54 54 52 89 89 85 F F F <--- 16 320-339 51 51 51 84 84 84 F F F 17 340-359 53 53 53 88 88 88 F F F 18 360-379 54 54 51 89 89 84 F F T <--- 19 380-399 54 54 53 89 89 88 F F T <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 49 89 89 81 F F F <--- 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 54 54 50 90 90 83 F F F <--- 27 540-559 54 54 48 90 90 79 F F F <--- 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 50 50 83 83 83 F F F Max gen: 54, Max flush: 90 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 83 83 83 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 54 54 54 89 89 89 F F F 3 060-079 54 54 54 89 89 89 F F F 4 080-099 53 53 53 88 88 88 F F F 5 100-119 55 55 55 91 91 91 F F F 6 120-139 55 55 55 91 91 91 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 50 50 50 82 82 82 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 55 55 55 91 91 91 F F F 12 240-259 53 53 53 88 88 88 F F F 13 260-279 54 54 54 89 89 89 F F F 14 280-299 50 50 50 82 82 82 F F F 15 300-319 54 54 54 89 89 89 F F F 16 320-339 51 51 51 84 84 84 F F F 17 340-359 53 53 53 88 88 88 F F F 18 360-379 54 54 54 89 89 89 F F F 19 380-399 54 54 54 89 89 89 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 54 54 54 90 90 90 F F F 27 540-559 54 54 54 90 90 90 F F F 28 560-579 43 43 43 66 66 66 F F F 29 580-599 50 50 50 83 83 83 F F F Max gen: 55, Max flush: 91 Begin loop 28 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.818129548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.829764264Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830113833Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83015222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830164436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830182224Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"Crucible 43be4772-8e6f-4cf1-9d21-5cc720be21a5 has session id: 8fa1655e-1536-401c-9631-876db2cb699e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830570331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Upstairs opts: Upstairs UUID: 43be4772-8e6f-4cf1-9d21-5cc720be21a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830601615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Crucible stats registered with UUID: 43be4772-8e6f-4cf1-9d21-5cc720be21a5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830617192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:42.830630188Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830796704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83083264Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830849578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83096326Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830980307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"43be4772-8e6f-4cf1-9d21-5cc720be21a5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.830995494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831012912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831026918Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831040695Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831054321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831083664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831137369Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39603)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831153576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"ds_connection connected from Ok(127.0.0.1:56555)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831168663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"ds_connection connected from Ok(127.0.0.1:62626)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831243927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"io task","client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831916123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.831960954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832034997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832319747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832364167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[0]R flush_numbers[0..12]: [83, 63, 89, 89, 88, 91, 91, 64, 89, 82, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832408717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[0]R generation[0..12]: [50, 41, 54, 54, 53, 55, 55, 42, 54, 50, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832425295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832441712Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[1]R flush_numbers[0..12]: [83, 63, 89, 89, 88, 91, 91, 64, 89, 82, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83245879Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[1]R generation[0..12]: [50, 41, 54, 54, 53, 55, 55, 42, 54, 50, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832474267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832490745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[2]R flush_numbers[0..12]: [83, 63, 89, 89, 88, 91, 91, 64, 89, 82, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832505852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[2]R generation[0..12]: [50, 41, 54, 54, 53, 55, 55, 42, 54, 50, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832520898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832537406Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832550632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:42.832564878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Next flush: 92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832584297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832597513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"":"downstairs","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83261163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832626476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"0","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832640583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"1","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832654579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"client":"2","":"downstairs","session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832669116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832685643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"43be4772-8e6f-4cf1-9d21-5cc720be21a5 is now active with session: 8fa1655e-1536-401c-9631-876db2cb699e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.83270088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832714336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426,"session_id":"8fa1655e-1536-401c-9631-876db2cb699e"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832727713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} {"msg":"Activated sub_volume 43be4772-8e6f-4cf1-9d21-5cc720be21a5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:42.832739638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 380 len 28672 data: 8 5 7 8 8 6 7 02/30 Write block 187 len 20480 data: 4 4 7 5 6 03/30 Read block 141 len 8192 04/30 Read block 305 len 20480 05/30 Read block 547 len 36864 06/30 Flush 07/30 Write block 261 len 12288 data: 5 4 5 08/30 Write block 344 len 24576 data: 4 4 5 4 6 6 09/30 Write block 564 len 12288 data: 3 4 5 10/30 Read block 491 len 12288 11/30 Read block 200 len 40960 12/30 Read block 427 len 36864 13/30 Write block 294 len 40960 data: 4 4 4 4 3 3 3 3 3 4 14/30 Flush 15/30 Write block 0 len 28672 data: 2 2 2 4 4 5 6 16/30 Read block 527 len 4096 17/30 Write block 291 len 16384 data: 7 7 5 5 18/30 Flush 19/30 Write block 588 len 12288 data: 4 5 5 20/30 Flush 21/30 Write block 131 len 16384 data: 7 5 6 9 22/30 Read block 159 len 40960 23/30 Read block 58 len 24576 24/30 Write block 528 len 12288 data: 3 4 4 25/30 Read block 396 len 20480 26/30 Write block 584 len 20480 data: 7 7 7 4 5 27/30 Write block 532 len 40960 data: 7 7 6 7 6 6 6 7 6 6 28/30 Read block 324 len 8192 29/30 Write block 592 len 32768 data: 5 5 6 6 6 5 4 3 30/30 Read block 162 len 28672 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 3 Done Done Done false 1007 Acked Write 6 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 23 1 24 Done 24 1 23 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1005)) Some(JobId(1019)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:42.845820121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1426} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 50 56 94 83 94 F F F <--- 1 020-039 41 41 41 63 63 63 F F F 2 040-059 54 54 54 89 89 89 F F F 3 060-079 54 54 54 89 89 89 F F F 4 080-099 53 53 53 88 88 88 F F F 5 100-119 55 55 55 91 91 91 F F F 6 120-139 55 55 55 91 91 91 T F T <--- 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 55 55 55 91 91 91 F F F 12 240-259 53 53 53 88 88 88 F F F 13 260-279 56 54 56 93 89 93 F T F <--- 14 280-299 56 50 56 94 82 94 F F F <--- 15 300-319 56 54 56 93 89 93 F F F <--- 16 320-339 51 51 51 84 84 84 F F F 17 340-359 56 53 56 93 88 93 F F F <--- 18 360-379 54 54 54 89 89 89 F F F 19 380-399 56 56 56 92 92 92 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 54 54 54 90 90 90 T F T <--- 27 540-559 54 54 54 90 90 90 T F T <--- 28 560-579 56 43 56 93 66 93 F F F <--- 29 580-599 56 50 56 95 83 95 T F T <--- Max gen: 56, Max flush: 95 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 54 54 54 89 89 89 F F F 3 060-079 54 54 54 89 89 89 F F F 4 080-099 53 53 53 88 88 88 F F F 5 100-119 55 55 55 91 91 91 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 48 48 48 78 78 78 F F F 11 220-239 55 55 55 91 91 91 F F F 12 240-259 53 53 53 88 88 88 F F F 13 260-279 56 56 56 93 93 93 F F F 14 280-299 56 56 56 94 94 94 F F F 15 300-319 56 56 56 93 93 93 F F F 16 320-339 51 51 51 84 84 84 F F F 17 340-359 56 56 56 93 93 93 F F F 18 360-379 54 54 54 89 89 89 F F F 19 380-399 56 56 56 92 92 92 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 47 47 47 76 76 76 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 51 51 51 84 84 84 F F F 26 520-539 57 57 57 96 96 96 F F F 27 540-559 57 57 57 96 96 96 F F F 28 560-579 56 56 56 93 93 93 F F F 29 580-599 57 57 57 96 96 96 F F F Max gen: 57, Max flush: 96 Begin loop 29 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.165895961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.177596736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.177915221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.177954779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.177967105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.177985353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"Crucible ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291 has session id: b5ae90a5-1072-4c75-b44b-9b64e20cb9ce","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178346837Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Upstairs opts: Upstairs UUID: ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178372999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Crucible stats registered with UUID: ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178390607Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:47.178403913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178565627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178589158Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178604044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178712214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178729622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178752882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178765938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178779945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178793351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178807037Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178821744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.17883589Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:51215)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178881941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:32954)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178897078Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:60401)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.178914366Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"io task","client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.179610062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.179638005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.179654743Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.179983792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180026862Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[0]R flush_numbers[0..12]: [94, 63, 89, 89, 88, 91, 96, 64, 89, 92, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180108219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[0]R generation[0..12]: [56, 41, 54, 54, 53, 55, 57, 42, 54, 56, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180125267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180141884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[1]R flush_numbers[0..12]: [94, 63, 89, 89, 88, 91, 96, 64, 89, 92, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180157071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[1]R generation[0..12]: [56, 41, 54, 54, 53, 55, 57, 42, 54, 56, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180172198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180188446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[2]R flush_numbers[0..12]: [94, 63, 89, 89, 88, 91, 96, 64, 89, 92, 78, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180203512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[2]R generation[0..12]: [56, 41, 54, 54, 53, 55, 57, 42, 54, 56, 48, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180218869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180238198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180251554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180271183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Next flush: 97","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180284759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180298136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"":"downstairs","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180312522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180325068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"0","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180339164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"1","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180353041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"client":"2","":"downstairs","session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"level":30,"time":"2026-03-14T05:02:47.180370578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180384645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291 is now active with session: b5ae90a5-1072-4c75-b44b-9b64e20cb9ce","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180399612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180412528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434,"session_id":"b5ae90a5-1072-4c75-b44b-9b64e20cb9ce"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.180425954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} {"msg":"Activated sub_volume ddc25b15-ac8e-46ae-bf1f-3fccfa0bf291","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:47.18044008Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 243 len 32768 data: 2 4 5 8 9 9 9 10 02/30 Write block 218 len 16384 data: 4 4 5 6 03/30 Read block 533 len 28672 04/30 Write block 369 len 24576 data: 7 7 7 8 8 10 05/30 Write block 432 len 8192 data: 7 6 06/30 Write block 365 len 4096 data: 5 07/30 Flush 08/30 Write block 529 len 4096 data: 5 09/30 Read block 559 len 8192 10/30 Write block 510 len 12288 data: 7 7 7 11/30 Read block 350 len 4096 12/30 Read block 540 len 20480 13/30 Read block 144 len 40960 14/30 Flush 15/30 Write block 372 len 16384 data: 9 9 11 8 16/30 Read block 220 len 16384 17/30 Write block 389 len 28672 data: 6 6 5 5 4 5 3 18/30 Write block 53 len 24576 data: 7 8 8 6 5 6 19/30 Read block 287 len 12288 20/30 Write block 315 len 16384 data: 6 7 6 7 21/30 Flush 22/30 Write block 292 len 12288 data: 8 6 6 23/30 Write block 593 len 20480 data: 6 7 7 7 6 24/30 Write block 76 len 20480 data: 6 8 8 8 8 25/30 Write block 430 len 40960 data: 8 8 8 7 5 5 6 5 7 7 26/30 Write block 106 len 32768 data: 3 3 5 3 4 4 4 5 27/30 Write block 341 len 8192 data: 6 4 28/30 Write block 381 len 16384 data: 6 8 9 9 29/30 Write block 235 len 12288 data: 6 6 5 30/30 Read block 376 len 36864 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 4 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:47.191027919Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1434} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 58 54 58 99 89 99 F F F <--- 3 060-079 54 54 54 89 89 89 T F T <--- 4 080-099 53 53 53 88 88 88 T F T <--- 5 100-119 55 55 55 91 91 91 T F T <--- 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 58 48 58 97 78 97 F F F <--- 11 220-239 58 55 58 97 91 97 T F T <--- 12 240-259 58 53 58 97 88 97 F T F <--- 13 260-279 56 56 56 93 93 93 F F F 14 280-299 56 56 56 94 94 94 T F T <--- 15 300-319 58 56 58 99 93 99 F F F <--- 16 320-339 51 51 51 84 84 84 F F F 17 340-359 56 56 56 93 93 93 T F T <--- 18 360-379 58 54 58 99 89 99 F F F <--- 19 380-399 58 56 58 99 92 99 T F T <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 58 47 58 97 76 97 T F T <--- 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 58 51 58 98 84 98 F F F <--- 26 520-539 58 57 58 98 96 98 F F F <--- 27 540-559 57 57 57 96 96 96 F F F 28 560-579 56 56 56 93 93 93 F F F 29 580-599 57 57 57 96 96 96 T F T <--- Max gen: 58, Max flush: 99 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 58 58 58 99 99 99 F F F 3 060-079 59 59 59 100 100 100 F F F 4 080-099 59 59 59 100 100 100 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 58 58 58 97 97 97 F F F 11 220-239 59 59 59 100 100 100 F F F 12 240-259 58 58 58 97 97 97 F F F 13 260-279 56 56 56 93 93 93 F F F 14 280-299 59 59 59 100 100 100 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 51 51 51 84 84 84 F F F 17 340-359 59 59 59 100 100 100 F F F 18 360-379 58 58 58 99 99 99 F F F 19 380-399 59 59 59 100 100 100 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 59 59 59 100 100 100 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 47 47 47 76 76 76 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 57 57 57 96 96 96 F F F 28 560-579 56 56 56 93 93 93 F F F 29 580-599 59 59 59 100 100 100 F F F Max gen: 59, Max flush: 100 Begin loop 30 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.527201994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.538834529Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539155115Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539192262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539204737Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539222645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"Crucible 29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e has session id: 9b3982f1-aafe-41c5-81bb-f7dc38679565","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539567883Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Upstairs opts: Upstairs UUID: 29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539613884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Crucible stats registered with UUID: 29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539629501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:51.539644988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.53982639Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539860946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539876983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539977149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.539991705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540030153Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54004638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540060977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540074813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54008867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540103076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540117633Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"ds_connection connected from Ok(127.0.0.1:56507)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540225682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"ds_connection connected from Ok(127.0.0.1:52812)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540356582Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"ds_connection connected from Ok(127.0.0.1:38236)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.5403757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"io task","client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540907462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.540930883Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54094699Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541261503Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541300371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[0]R flush_numbers[0..12]: [94, 63, 99, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541361769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[0]R generation[0..12]: [56, 41, 58, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541381498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541398266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[1]R flush_numbers[0..12]: [94, 63, 99, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541413863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[1]R generation[0..12]: [56, 41, 58, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54142927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541445567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[2]R flush_numbers[0..12]: [94, 63, 99, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541460874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[2]R generation[0..12]: [56, 41, 58, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541476081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541492629Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:51.541506105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541526164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Next flush: 101","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.5415399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541553366Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"":"downstairs","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541567923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541580329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"0","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541596116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541613534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"client":"2","":"downstairs","session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54162817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541641847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e is now active with session: 9b3982f1-aafe-41c5-81bb-f7dc38679565","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541656723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541669389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442,"session_id":"9b3982f1-aafe-41c5-81bb-f7dc38679565"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.541682865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} {"msg":"Activated sub_volume 29e2fa6e-8d67-4cf4-bde7-5ee2ed97c11e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:51.54169448Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 489 len 24576 data: 5 5 5 5 7 9 02/30 Read block 153 len 32768 03/30 Read block 350 len 20480 04/30 Read block 486 len 24576 05/30 Read block 508 len 8192 06/30 Write block 322 len 28672 data: 8 7 7 7 9 8 8 07/30 Read block 148 len 8192 08/30 Read block 585 len 20480 09/30 Read block 260 len 8192 10/30 Write block 580 len 24576 data: 9 10 8 7 8 8 11/30 Write block 269 len 28672 data: 6 5 7 7 8 8 8 12/30 Write block 48 len 12288 data: 4 5 5 13/30 Flush 14/30 Write block 343 len 12288 data: 5 5 5 15/30 Read block 155 len 28672 16/30 Write block 257 len 4096 data: 6 17/30 Read block 533 len 16384 18/30 Read block 88 len 28672 19/30 Write block 289 len 4096 data: 9 20/30 Read block 103 len 28672 21/30 Read block 514 len 36864 22/30 Write block 437 len 12288 data: 6 8 8 23/30 Read block 450 len 20480 24/30 Write block 377 len 16384 data: 7 6 8 9 25/30 Read block 170 len 12288 26/30 Read block 152 len 20480 27/30 Read block 59 len 8192 28/30 Write block 530 len 24576 data: 5 6 8 8 7 8 29/30 Read block 201 len 12288 30/30 Write block 47 len 28672 data: 4 5 6 6 6 5 8 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:51.553165011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1442} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 60 58 60 101 99 101 T F T <--- 3 060-079 59 59 59 100 100 100 F F F 4 080-099 59 59 59 100 100 100 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 58 58 58 97 97 97 F F F 11 220-239 59 59 59 100 100 100 F F F 12 240-259 58 58 58 97 97 97 T F T <--- 13 260-279 60 56 60 101 93 101 F F F <--- 14 280-299 59 59 59 100 100 100 T F T <--- 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 51 60 101 84 101 F F F <--- 17 340-359 59 59 59 100 100 100 T F T <--- 18 360-379 58 58 58 99 99 99 T F T <--- 19 380-399 59 59 59 100 100 100 T F T <--- 20 400-419 45 45 45 68 68 68 F F F 21 420-439 59 59 59 100 100 100 T F T <--- 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 47 60 101 76 101 F T F <--- 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 T F T <--- 27 540-559 57 57 57 96 96 96 F F F 28 560-579 56 56 56 93 93 93 F F F 29 580-599 60 59 60 101 100 101 F F F <--- Max gen: 60, Max flush: 101 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 41 41 63 63 63 F F F 2 040-059 61 61 61 102 102 102 F F F 3 060-079 59 59 59 100 100 100 F F F 4 080-099 59 59 59 100 100 100 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F F F 9 180-199 56 56 56 92 92 92 F F F 10 200-219 58 58 58 97 97 97 F F F 11 220-239 59 59 59 100 100 100 F F F 12 240-259 61 61 61 102 102 102 F F F 13 260-279 60 60 60 101 101 101 F F F 14 280-299 61 61 61 102 102 102 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 61 61 61 102 102 102 F F F 18 360-379 61 61 61 102 102 102 F F F 19 380-399 61 61 61 102 102 102 F F F 20 400-419 45 45 45 68 68 68 F F F 21 420-439 61 61 61 102 102 102 F F F 22 440-459 53 53 53 88 88 88 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 60 60 101 101 101 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 61 61 61 102 102 102 F F F 27 540-559 57 57 57 96 96 96 F F F 28 560-579 56 56 56 93 93 93 F F F 29 580-599 60 60 60 101 101 101 F F F Max gen: 61, Max flush: 102 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.880053949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:55.890695923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.890749787Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901149981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901441803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901480881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901492957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901510695Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"Crucible 5f35510f-360e-4bde-88e9-91875d40b51b has session id: 7b7b8009-ab51-4187-9181-88eb8a7f3918","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901852981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Upstairs opts: Upstairs UUID: 5f35510f-360e-4bde-88e9-91875d40b51b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901882424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Crucible stats registered with UUID: 5f35510f-360e-4bde-88e9-91875d40b51b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.901897421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:55.901910467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902085546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902109857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902124724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902245789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902260776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"5f35510f-360e-4bde-88e9-91875d40b51b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902281786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902294672Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902308588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902322584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.90233623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902350667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902365134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"ds_connection connected from Ok(127.0.0.1:58683)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902392396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"ds_connection connected from Ok(127.0.0.1:52086)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.902407763Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"ds_connection connected from Ok(127.0.0.1:64955)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.90242274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"io task","client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903195381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903223584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903265393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903591902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903605958Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[0]R flush_numbers[0..12]: [94, 63, 102, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903702102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[0]R generation[0..12]: [56, 41, 61, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.90371883Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903735547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[1]R flush_numbers[0..12]: [94, 63, 102, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903750644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[1]R generation[0..12]: [56, 41, 61, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903765921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903782429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[2]R flush_numbers[0..12]: [94, 63, 102, 100, 100, 100, 96, 64, 89, 92, 97, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903801277Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[2]R generation[0..12]: [56, 41, 61, 59, 59, 59, 57, 42, 54, 56, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903816354Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903832441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903845858Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903865396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Next flush: 103","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903878923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903892359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"":"downstairs","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903906865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903919731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"0","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903933447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"1","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :30,"time":"2026-03-14T05:02:55.903950015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"client":"2","":"downstairs","session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903964151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903977387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"5f35510f-360e-4bde-88e9-91875d40b51b is now active with session: 7b7b8009-ab51-4187-9181-88eb8a7f3918","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.903992334Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.90400512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450,"session_id":"7b7b8009-ab51-4187-9181-88eb8a7f3918"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.904018476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} {"msg":"Activated sub_volume 5f35510f-360e-4bde-88e9-91875d40b51b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:55.904031732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 90 len 20480 data: 9 10 10 9 8 03/30 Write block 351 len 36864 data: 7 6 6 6 5 4 4 6 4 04/30 Write block 205 len 12288 data: 2 3 3 05/30 Write block 30 len 36864 data: 4 5 5 5 5 6 6 5 6 06/30 Write block 396 len 28672 data: 4 4 4 4 4 4 3 07/30 Read block 140 len 8192 08/30 Write block 409 len 20480 data: 6 6 7 7 6 09/30 Read block 564 len 28672 10/30 Read block 484 len 36864 11/30 Read block 538 len 4096 12/30 Read block 305 len 16384 13/30 Read block 245 len 16384 14/30 Read block 456 len 40960 15/30 Write block 547 len 16384 data: 5 6 4 5 16/30 Write block 182 len 20480 data: 6 6 5 5 5 17/30 Flush 18/30 Flush 19/30 Write block 412 len 24576 data: 8 7 5 6 5 4 20/30 Write block 262 len 24576 data: 5 6 5 5 6 6 21/30 Write block 175 len 16384 data: 4 5 5 5 22/30 Write block 539 len 12288 data: 8 7 7 23/30 Write block 240 len 40960 data: 2 2 2 3 5 6 9 10 10 10 24/30 Write block 438 len 12288 data: 9 9 6 25/30 Read block 368 len 24576 26/30 Read block 344 len 16384 27/30 Write block 504 len 12288 data: 7 8 6 28/30 Write block 563 len 28672 data: 2 4 5 6 5 7 10 29/30 Read block 475 len 4096 30/30 Write block 68 len 32768 data: 7 6 5 4 4 5 5 7 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 2 Done Done Done false 1007 Acked Write 5 Done Done Done false 1008 Acked Read 7 Done Done Done false 1009 Acked Read 9 Done Done Done false 1010 Acked Read 1 Done Done Done false 1011 Acked Read 4 Done Done Done false 1012 Acked Read 4 Done Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 1 1 19 Done 12 28 28 68 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:55.91516671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1450} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 41 62 62 63 104 104 T F F <--- 2 040-059 61 61 61 102 102 102 F F F 3 060-079 59 59 59 100 100 100 F T T <--- 4 080-099 59 62 62 100 104 104 T F F <--- 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 54 54 54 89 89 89 F T T <--- 9 180-199 56 62 62 92 104 104 F F F <--- 10 200-219 58 62 62 97 104 104 T F F <--- 11 220-239 59 59 59 100 100 100 F F F 12 240-259 61 61 61 102 102 102 F T T <--- 13 260-279 60 60 60 101 101 101 F T T <--- 14 280-299 61 61 61 102 102 102 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 61 62 62 102 104 104 T F F <--- 18 360-379 61 61 61 102 102 102 F F F 19 380-399 61 62 62 102 104 104 T F F <--- 20 400-419 45 62 62 68 104 104 T T T <--- 21 420-439 61 61 61 102 102 102 F T T <--- 22 440-459 53 53 53 88 88 88 F T T <--- 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 60 60 101 101 101 F F F 25 500-519 58 58 58 98 98 98 F T T <--- 26 520-539 61 61 61 102 102 102 F T T <--- 27 540-559 57 62 62 96 104 104 F T T <--- 28 560-579 56 56 56 93 93 93 F T T <--- 29 580-599 60 60 60 101 101 101 F F F Max gen: 62, Max flush: 104 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 94 94 94 F F F 1 020-039 62 62 62 104 104 104 F F F 2 040-059 61 61 61 102 102 102 F F F 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 63 63 63 105 105 105 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 59 59 59 100 100 100 F F F 12 240-259 63 63 63 105 105 105 F F F 13 260-279 63 63 63 105 105 105 F F F 14 280-299 61 61 61 102 102 102 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 62 62 104 104 104 F F F 18 360-379 61 61 61 102 102 102 F F F 19 380-399 62 62 62 104 104 104 F F F 20 400-419 63 63 63 105 105 105 F F F 21 420-439 63 63 63 105 105 105 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 60 60 101 101 101 F F F 25 500-519 63 63 63 105 105 105 F F F 26 520-539 63 63 63 105 105 105 F F F 27 540-559 63 63 63 105 105 105 F F F 28 560-579 63 63 63 105 105 105 F F F 29 580-599 60 60 60 101 101 101 F F F Max gen: 63, Max flush: 105 Begin loop 32 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.263429869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:00.274083288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.274121155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.284341717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.284628477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.284666764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.28467923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.284697679Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"Crucible 24662fe9-c55b-4419-a0c0-0160095c9df1 has session id: 1cf43770-e3a9-4ff4-b338-c85347fa0380","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.28505174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Upstairs opts: Upstairs UUID: 24662fe9-c55b-4419-a0c0-0160095c9df1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285085795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Crucible stats registered with UUID: 24662fe9-c55b-4419-a0c0-0160095c9df1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285101622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:00.285115299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285266668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285301814Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285321312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285435925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285451091Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"24662fe9-c55b-4419-a0c0-0160095c9df1 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285468719Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285481215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285495422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285509528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285548276Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285581381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285596818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"ds_connection connected from Ok(127.0.0.1:64253)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285618738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"ds_connection connected from Ok(127.0.0.1:64222)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285633635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"ds_connection connected from Ok(127.0.0.1:49497)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.285731419Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"io task","client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286416091Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286453668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286506932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286785429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286824106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[0]R flush_numbers[0..12]: [94, 104, 102, 105, 104, 100, 96, 64, 105, 104, 104, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286888015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[0]R generation[0..12]: [56, 62, 61, 63, 62, 59, 57, 42, 63, 62, 62, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286904843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286921211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[1]R flush_numbers[0..12]: [94, 104, 102, 105, 104, 100, 96, 64, 105, 104, 104, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286936868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[1]R generation[0..12]: [56, 62, 61, 63, 62, 59, 57, 42, 63, 62, 62, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286952355Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286968462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[2]R flush_numbers[0..12]: [94, 104, 102, 105, 104, 100, 96, 64, 105, 104, 104, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286983839Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[2]R generation[0..12]: [56, 62, 61, 63, 62, 59, 57, 42, 63, 62, 62, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.286999336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287015914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.2870293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287050379Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Next flush: 106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287065846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287080023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"":"downstairs","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.28709477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287107535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"0","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287121622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"1","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287135488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"client":"2","":"downstairs","session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287152156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287166142Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"24662fe9-c55b-4419-a0c0-0160095c9df1 is now active with session: 1cf43770-e3a9-4ff4-b338-c85347fa0380","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287181249Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287193825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458,"session_id":"1cf43770-e3a9-4ff4-b338-c85347fa0380"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287207611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} {"msg":"Activated sub_volume 24662fe9-c55b-4419-a0c0-0160095c9df1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:00.287219066Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 267 len 16384 data: 7 6 7 6 02/30 Write block 5 len 36864 data: 6 7 5 7 7 6 6 7 6 03/30 Read block 534 len 32768 04/30 Write block 230 len 28672 data: 9 10 8 6 7 7 7 05/30 Flush 06/30 Write block 575 len 4096 data: 7 07/30 Read block 435 len 4096 08/30 Write block 30 len 20480 data: 5 6 6 6 6 09/30 Read block 239 len 20480 10/30 Read block 460 len 20480 11/30 Read block 211 len 24576 12/30 Read block 227 len 16384 13/30 Write block 375 len 36864 data: 9 8 8 7 9 10 7 9 10 14/30 Read block 370 len 40960 15/30 Read block 351 len 28672 16/30 Write block 287 len 40960 data: 8 8 10 8 8 9 7 7 5 5 17/30 Read block 178 len 32768 18/30 Write block 384 len 24576 data: 10 7 8 8 7 7 19/30 Write block 248 len 32768 data: 11 11 11 9 10 8 10 10 20/30 Read block 84 len 36864 21/30 Write block 563 len 8192 data: 3 5 22/30 Write block 420 len 20480 data: 3 2 4 4 6 23/30 Read block 95 len 12288 24/30 Write block 167 len 32768 data: 5 5 5 6 6 6 5 5 25/30 Read block 20 len 40960 26/30 Write block 8 len 24576 data: 8 8 7 7 8 7 27/30 Write block 131 len 32768 data: 8 6 7 10 10 10 7 7 28/30 Write block 42 len 28672 data: 6 4 5 4 4 5 6 29/30 Read block 156 len 20480 30/30 Read block 103 len 32768 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:00.300418604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1458} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 64 64 94 106 106 F T T <--- 1 020-039 62 62 62 104 104 104 F T T <--- 2 040-059 61 61 61 102 102 102 F T T <--- 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 57 57 57 96 96 96 F T T <--- 7 140-159 42 42 42 64 64 64 F F F 8 160-179 63 63 63 105 105 105 F T T <--- 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 59 64 64 100 106 106 F F F <--- 12 240-259 63 63 63 105 105 105 F T T <--- 13 260-279 63 64 64 105 106 106 T F F <--- 14 280-299 61 61 61 102 102 102 F T T <--- 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 62 62 104 104 104 F F F 18 360-379 61 61 61 102 102 102 F T T <--- 19 380-399 62 62 62 104 104 104 F T T <--- 20 400-419 63 63 63 105 105 105 F F F 21 420-439 63 63 63 105 105 105 F T T <--- 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 60 60 101 101 101 F F F 25 500-519 63 63 63 105 105 105 F F F 26 520-539 63 63 63 105 105 105 F F F 27 540-559 63 63 63 105 105 105 F F F 28 560-579 63 63 63 105 105 105 F T T <--- 29 580-599 60 60 60 101 101 101 F F F Max gen: 64, Max flush: 106 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 65 65 65 107 107 107 F F F 2 040-059 65 65 65 107 107 107 F F F 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F F F 5 100-119 59 59 59 100 100 100 F F F 6 120-139 65 65 65 107 107 107 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 65 65 65 107 107 107 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 64 64 106 106 106 F F F 12 240-259 65 65 65 107 107 107 F F F 13 260-279 64 64 64 106 106 106 F F F 14 280-299 65 65 65 107 107 107 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 62 62 104 104 104 F F F 18 360-379 65 65 65 107 107 107 F F F 19 380-399 65 65 65 107 107 107 F F F 20 400-419 63 63 63 105 105 105 F F F 21 420-439 65 65 65 107 107 107 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 60 60 101 101 101 F F F 25 500-519 63 63 63 105 105 105 F F F 26 520-539 63 63 63 105 105 105 F F F 27 540-559 63 63 63 105 105 105 F F F 28 560-579 65 65 65 107 107 107 F F F 29 580-599 60 60 60 101 101 101 F F F Max gen: 65, Max flush: 107 Begin loop 33 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.640405975Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:04.651039415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.651092489Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.6613342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.661638198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.661676726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.661688862Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.6617066Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"Crucible 7d866b9b-7568-4869-9a41-08ea15b6afe1 has session id: 4aad5738-4904-4796-9329-b1234de13780","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662072196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Upstairs opts: Upstairs UUID: 7d866b9b-7568-4869-9a41-08ea15b6afe1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662107852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Crucible stats registered with UUID: 7d866b9b-7568-4869-9a41-08ea15b6afe1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.66212405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:04.662137956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662282632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662317588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662333325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662437092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662451909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"7d866b9b-7568-4869-9a41-08ea15b6afe1 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662469187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662481872Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.66249903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662513357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662526903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662541009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662555216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"ds_connection connected from Ok(127.0.0.1:47072)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662592483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"ds_connection connected from Ok(127.0.0.1:38875)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.66260856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"ds_connection connected from Ok(127.0.0.1:56424)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.662670568Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"io task","client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663270762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663339353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663367025Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663671554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663715904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[0]R flush_numbers[0..12]: [107, 107, 107, 105, 104, 100, 107, 64, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663780534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[0]R generation[0..12]: [65, 65, 65, 63, 62, 59, 65, 42, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663797391Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.6638167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[1]R flush_numbers[0..12]: [107, 107, 107, 105, 104, 100, 107, 64, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663832667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[1]R generation[0..12]: [65, 65, 65, 63, 62, 59, 65, 42, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663848054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663864262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[2]R flush_numbers[0..12]: [107, 107, 107, 105, 104, 100, 107, 64, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663879999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[2]R generation[0..12]: [65, 65, 65, 63, 62, 59, 65, 42, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663895396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663911583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Max found gen is 66","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-03-14T05:03:04.663930832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663944918Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Next flush: 108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663958525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663972071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"":"downstairs","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663986547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.663999383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"0","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.66401382Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"1","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :30,"time":"2026-03-14T05:03:04.664033078Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"client":"2","":"downstairs","session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.664047905Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.664061131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"7d866b9b-7568-4869-9a41-08ea15b6afe1 is now active with session: 4aad5738-4904-4796-9329-b1234de13780","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.664076088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.664088634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466,"session_id":"4aad5738-4904-4796-9329-b1234de13780"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.66410172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} {"msg":"Activated sub_volume 7d866b9b-7568-4869-9a41-08ea15b6afe1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:04.664112865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 577 len 32768 data: 7 6 7 10 11 9 8 9 02/30 Write block 135 len 40960 data: 11 11 8 8 6 5 3 3 5 5 03/30 Read block 485 len 40960 04/30 Write block 550 len 4096 data: 6 05/30 Flush 06/30 Write block 46 len 12288 data: 5 6 7 07/30 Write block 119 len 36864 data: 4 4 5 6 5 5 5 6 7 08/30 Flush 09/30 Flush 10/30 Write block 388 len 32768 data: 8 8 7 6 6 5 6 4 11/30 Write block 33 len 8192 data: 7 7 12/30 Read block 358 len 36864 13/30 Read block 68 len 4096 14/30 Write block 361 len 40960 data: 4 5 4 6 6 4 5 7 8 8 15/30 Read block 355 len 4096 16/30 Write block 45 len 28672 data: 5 6 7 8 7 7 7 17/30 Read block 236 len 36864 18/30 Flush 19/30 Write block 289 len 24576 data: 11 9 9 10 8 8 20/30 Write block 575 len 20480 data: 8 6 8 7 8 21/30 Write block 498 len 32768 data: 9 9 9 6 5 7 8 9 22/30 Write block 583 len 24576 data: 9 10 9 8 5 6 23/30 Read block 526 len 8192 24/30 Read block 541 len 16384 25/30 Read block 248 len 40960 26/30 Flush 27/30 Write block 108 len 20480 data: 6 4 5 5 5 28/30 Read block 544 len 24576 29/30 Read block 452 len 12288 30/30 Write block 565 len 40960 data: 6 7 6 8 11 9 8 7 5 8 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:04.677002171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1466} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 65 66 66 107 111 111 F F F <--- 2 040-059 65 66 66 107 111 111 F F F <--- 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F F F 5 100-119 59 66 66 100 109 109 F T T <--- 6 120-139 65 66 66 107 109 109 F F F <--- 7 140-159 42 66 66 64 108 108 F F F <--- 8 160-179 65 65 65 107 107 107 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 64 64 106 106 106 F F F 12 240-259 65 65 65 107 107 107 F F F 13 260-279 64 64 64 106 106 106 F F F 14 280-299 65 66 66 107 112 112 F F F <--- 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 62 62 104 104 104 F F F 18 360-379 65 66 66 107 111 111 F F F <--- 19 380-399 65 66 66 107 111 111 F F F <--- 20 400-419 63 63 63 105 105 105 F F F 21 420-439 65 65 65 107 107 107 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 60 66 66 101 112 112 F F F <--- 25 500-519 63 66 66 105 112 112 F F F <--- 26 520-539 63 63 63 105 105 105 F F F 27 540-559 63 66 66 105 108 108 F F F <--- 28 560-579 65 66 66 107 112 112 F T T <--- 29 580-599 60 66 66 101 112 112 F F F <--- Max gen: 66, Max flush: 112 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 66 66 66 111 111 111 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F F F 5 100-119 67 67 67 113 113 113 F F F 6 120-139 66 66 66 109 109 109 F F F 7 140-159 66 66 66 108 108 108 F F F 8 160-179 65 65 65 107 107 107 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 64 64 106 106 106 F F F 12 240-259 65 65 65 107 107 107 F F F 13 260-279 64 64 64 106 106 106 F F F 14 280-299 66 66 66 112 112 112 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 62 62 104 104 104 F F F 18 360-379 66 66 66 111 111 111 F F F 19 380-399 66 66 66 111 111 111 F F F 20 400-419 63 63 63 105 105 105 F F F 21 420-439 65 65 65 107 107 107 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 66 66 66 112 112 112 F F F 26 520-539 63 63 63 105 105 105 F F F 27 540-559 66 66 66 108 108 108 F F F 28 560-579 67 67 67 113 113 113 F F F 29 580-599 66 66 66 112 112 112 F F F Max gen: 67, Max flush: 113 Begin loop 34 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.012339185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:09.023088337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.023145333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.033461519Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.033782735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.033823233Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.033835109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.033852577Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"Crucible d07237eb-78c2-4dfb-a110-293f23b6f3c4 has session id: c6671dae-73b9-4ef8-bec5-1cf11f06de13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034196213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Upstairs opts: Upstairs UUID: d07237eb-78c2-4dfb-a110-293f23b6f3c4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.03423279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Crucible stats registered with UUID: d07237eb-78c2-4dfb-a110-293f23b6f3c4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034248957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:09.034262293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.03441007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034434291Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034450419Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034574805Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034591163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"d07237eb-78c2-4dfb-a110-293f23b6f3c4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.03460755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034619996Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034634162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034648239Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034662175Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034676722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034691198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"ds_connection connected from Ok(127.0.0.1:49914)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034731236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"ds_connection connected from Ok(127.0.0.1:63265)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034755948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"ds_connection connected from Ok(127.0.0.1:48479)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.034770894Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"io task","client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035499416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035525338Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035541705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035861201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035906982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[0]R flush_numbers[0..12]: [107, 111, 111, 105, 104, 113, 109, 108, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035945559Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[0]R generation[0..12]: [65, 66, 66, 63, 62, 67, 66, 66, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035962227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035978604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[1]R flush_numbers[0..12]: [107, 111, 111, 105, 104, 113, 109, 108, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.035994111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[1]R generation[0..12]: [65, 66, 66, 63, 62, 67, 66, 66, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036009348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036025626Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[2]R flush_numbers[0..12]: [107, 111, 111, 105, 104, 113, 109, 108, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036044334Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[2]R generation[0..12]: [65, 66, 66, 63, 62, 67, 66, 66, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036060291Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036076209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"Wait for a query_work_queue command to finish before sending IO msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036096308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036110804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Next flush: 114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.03612396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036137576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"":"downstairs","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036152013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036164709Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"0","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036178945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"1","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Transition from WaitQuorum to Active","v":0,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 name":"crucible","level":30,"time":"2026-03-14T05:03:09.036195593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"client":"2","":"downstairs","session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036209959Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036223566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"d07237eb-78c2-4dfb-a110-293f23b6f3c4 is now active with session: c6671dae-73b9-4ef8-bec5-1cf11f06de13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036238472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036251078Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474,"session_id":"c6671dae-73b9-4ef8-bec5-1cf11f06de13"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.036264584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} {"msg":"Activated sub_volume d07237eb-78c2-4dfb-a110-293f23b6f3c4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:09.03627796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 249 len 4096 data: 12 02/30 Read block 134 len 32768 03/30 Read block 473 len 16384 04/30 Write block 243 len 32768 data: 4 6 7 10 11 12 13 12 05/30 Write block 130 len 20480 data: 7 9 7 8 11 06/30 Write block 512 len 36864 data: 8 7 8 6 5 5 4 3 3 07/30 Flush 08/30 Write block 393 len 36864 data: 6 7 5 5 5 5 5 5 5 09/30 Write block 149 len 32768 data: 4 5 6 5 4 6 5 5 10/30 Flush 11/30 Write block 124 len 24576 data: 6 6 7 8 6 6 12/30 Read block 252 len 32768 13/30 Write block 29 len 12288 data: 4 6 7 14/30 Read block 225 len 40960 15/30 Read block 283 len 32768 16/30 Read block 412 len 12288 17/30 Write block 345 len 24576 data: 6 6 5 7 7 5 18/30 Read block 435 len 8192 19/30 Flush 20/30 Write block 428 len 40960 data: 8 8 9 9 9 8 6 6 7 7 21/30 Read block 590 len 36864 22/30 Read block 332 len 4096 23/30 Write block 529 len 28672 data: 6 6 7 9 9 8 9 24/30 Write block 355 len 24576 data: 6 5 5 7 5 5 25/30 Read block 152 len 8192 26/30 Write block 131 len 4096 data: 10 27/30 Read block 571 len 8192 28/30 Write block 262 len 12288 data: 6 7 6 29/30 Write block 539 len 4096 data: 9 30/30 Write block 89 len 16384 data: 7 10 11 11 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 3 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:09.048005788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1474} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 66 68 68 111 116 116 F F F <--- 2 040-059 66 66 66 111 111 111 F F F 3 060-079 63 63 63 105 105 105 F F F 4 080-099 62 62 62 104 104 104 F T T <--- 5 100-119 67 67 67 113 113 113 F F F 6 120-139 66 68 68 109 116 116 F T T <--- 7 140-159 66 68 68 108 115 115 F F F <--- 8 160-179 65 65 65 107 107 107 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 64 64 106 106 106 F F F 12 240-259 65 68 68 107 114 114 F F F <--- 13 260-279 64 64 64 106 106 106 F T T <--- 14 280-299 66 66 66 112 112 112 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 62 68 68 104 116 116 F T T <--- 18 360-379 66 66 66 111 111 111 F T T <--- 19 380-399 66 68 68 111 115 115 F F F <--- 20 400-419 63 68 68 105 115 115 F F F <--- 21 420-439 65 65 65 107 107 107 F T T <--- 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 66 68 68 112 114 114 F F F <--- 26 520-539 63 68 68 105 114 114 F T T <--- 27 540-559 66 66 66 108 108 108 F F F 28 560-579 67 67 67 113 113 113 F F F 29 580-599 66 66 66 112 112 112 F F F Max gen: 68, Max flush: 116 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 63 63 63 105 105 105 F F F 4 080-099 69 69 69 117 117 117 F F F 5 100-119 67 67 67 113 113 113 F F F 6 120-139 69 69 69 117 117 117 F F F 7 140-159 68 68 68 115 115 115 F F F 8 160-179 65 65 65 107 107 107 F F F 9 180-199 62 62 62 104 104 104 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 64 64 106 106 106 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 69 69 117 117 117 F F F 14 280-299 66 66 66 112 112 112 F F F 15 300-319 58 58 58 99 99 99 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 69 69 69 117 117 117 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 68 68 68 115 115 115 F F F 20 400-419 68 68 68 115 115 115 F F F 21 420-439 69 69 69 117 117 117 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 68 68 68 114 114 114 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 66 66 66 108 108 108 F F F 28 560-579 67 67 67 113 113 113 F F F 29 580-599 66 66 66 112 112 112 F F F Max gen: 69, Max flush: 117 Begin loop 35 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.386426575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:13.39707189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.397125395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.407327098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.407613558Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.407651966Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.407664261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.407681979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"Crucible 965baa52-ac23-4d8b-be34-fbb9491da6fb has session id: 992b5f3e-0e9a-41f5-9551-f30afbc2631a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408036221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Upstairs opts: Upstairs UUID: 965baa52-ac23-4d8b-be34-fbb9491da6fb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408070496Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Crucible stats registered with UUID: 965baa52-ac23-4d8b-be34-fbb9491da6fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408086193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:13.40809961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.40827577Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408313797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"2","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408330345Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408439174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.40845268Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"965baa52-ac23-4d8b-be34-fbb9491da6fb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.40847401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408487046Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408501122Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408515039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"2","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408529015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"2","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408543512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408557888Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"ds_connection connected from Ok(127.0.0.1:48761)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408572495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"2","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41617)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408587322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"ds_connection connected from Ok(127.0.0.1:40893)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.408602258Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"io task","client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409290752Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"2","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409319205Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409339424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409667843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409709842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[0]R flush_numbers[0..12]: [107, 116, 111, 105, 117, 113, 117, 115, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409779944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[0]R generation[0..12]: [65, 68, 66, 63, 69, 67, 69, 68, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409796512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.40981332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[1]R flush_numbers[0..12]: [107, 116, 111, 105, 117, 113, 117, 115, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409829447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[1]R generation[0..12]: [65, 68, 66, 63, 69, 67, 69, 68, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409844704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409860721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[2]R flush_numbers[0..12]: [107, 116, 111, 105, 117, 113, 117, 115, 107, 104, 104, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409876428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[2]R generation[0..12]: [65, 68, 66, 63, 69, 67, 69, 68, 65, 62, 62, 64]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409891495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409907652Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409920949Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409940928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Next flush: 118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409956064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409969451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"":"downstairs","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409983767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.409996303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"0","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410010529Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"1","":"downstairs","session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410024586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410041293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410054879Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"965baa52-ac23-4d8b-be34-fbb9491da6fb is now active with session: 992b5f3e-0e9a-41f5-9551-f30afbc2631a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410069876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410082502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482,"session_id":"992b5f3e-0e9a-41f5-9551-f30afbc2631a"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410095758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} {"msg":"Activated sub_volume 965baa52-ac23-4d8b-be34-fbb9491da6fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:13.410107243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 290 len 12288 data: 10 10 11 02/30 Write block 73 len 36864 data: 6 6 8 7 9 9 9 9 6 03/30 Write block 148 len 4096 data: 4 04/30 Write block 152 len 8192 data: 6 5 05/30 Read block 50 len 40960 06/30 Write block 506 len 36864 data: 7 8 7 6 8 8 9 8 9 07/30 Read block 559 len 32768 08/30 Write block 307 len 20480 data: 4 4 5 6 7 09/30 Flush 10/30 Write block 224 len 8192 data: 8 9 11/30 Write block 122 len 40960 data: 7 6 7 7 8 9 7 7 8 11 12/30 Flush 13/30 Write block 548 len 28672 data: 7 5 7 6 6 9 8 14/30 Read block 320 len 36864 15/30 Write block 69 len 16384 data: 7 6 5 5 16/30 Write block 162 len 8192 data: 3 5 17/30 Read block 450 len 8192 18/30 Read block 445 len 24576 19/30 Flush 20/30 Write block 382 len 32768 data: 10 11 11 8 9 9 9 9 21/30 Read block 223 len 12288 22/30 Flush 23/30 Write block 549 len 32768 data: 6 8 7 7 10 9 8 7 24/30 Write block 124 len 28672 data: 8 8 9 10 8 8 9 25/30 Read block 284 len 8192 26/30 Read block 20 len 12288 27/30 Write block 69 len 32768 data: 8 7 6 6 7 7 9 8 28/30 Read block 177 len 40960 29/30 Write block 177 len 20480 data: 6 6 5 6 6 30/30 Read block 149 len 12288 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 3 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:13.422933081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1482} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 63 70 70 105 120 120 F T T <--- 4 080-099 69 70 70 117 118 118 F F F <--- 5 100-119 67 67 67 113 113 113 F F F 6 120-139 69 70 70 117 119 119 F T T <--- 7 140-159 68 70 70 115 118 118 F F F <--- 8 160-179 65 70 70 107 120 120 F T T <--- 9 180-199 62 62 62 104 104 104 F T T <--- 10 200-219 62 62 62 104 104 104 F F F 11 220-239 64 70 70 106 119 119 F F F <--- 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 69 69 117 117 117 F F F 14 280-299 66 70 70 112 118 118 F F F <--- 15 300-319 58 70 70 99 118 118 F F F <--- 16 320-339 60 60 60 101 101 101 F F F 17 340-359 69 69 69 117 117 117 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 68 70 70 115 121 121 F F F <--- 20 400-419 68 68 68 115 115 115 F F F 21 420-439 69 69 69 117 117 117 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 68 70 70 114 118 118 F F F <--- 26 520-539 69 69 69 117 117 117 F F F 27 540-559 66 70 70 108 120 120 F T T <--- 28 560-579 67 67 67 113 113 113 F F F 29 580-599 66 66 66 112 112 112 F F F Max gen: 70, Max flush: 121 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 107 107 107 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 71 71 71 122 122 122 F F F 4 080-099 70 70 70 118 118 118 F F F 5 100-119 67 67 67 113 113 113 F F F 6 120-139 71 71 71 122 122 122 F F F 7 140-159 70 70 70 118 118 118 F F F 8 160-179 71 71 71 122 122 122 F F F 9 180-199 71 71 71 122 122 122 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 70 70 70 119 119 119 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 69 69 117 117 117 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 70 70 70 118 118 118 F F F 16 320-339 60 60 60 101 101 101 F F F 17 340-359 69 69 69 117 117 117 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 70 70 70 121 121 121 F F F 20 400-419 68 68 68 115 115 115 F F F 21 420-439 69 69 69 117 117 117 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 54 54 89 89 89 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 70 70 70 118 118 118 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 71 71 71 122 122 122 F F F 28 560-579 67 67 67 113 113 113 F F F 29 580-599 66 66 66 112 112 112 F F F Max gen: 71, Max flush: 122 Begin loop 36 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.759333229Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:17.769914615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.769959756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780159748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780474791Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780514199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780526465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780544333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"Crucible c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d has session id: d79cc885-993d-4f86-8a90-8d2ca1dcac5e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780898624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Upstairs opts: Upstairs UUID: c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.78093359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Crucible stats registered with UUID: c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.780949537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:17.780962913Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781099155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781125728Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781140684Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781244612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.78126222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781295055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781308791Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781325759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781339705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781353441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781367708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781381934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:37176)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781435399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61255)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781529031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:50393)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.781598833Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"io task","client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782189972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782225518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782264746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782578559Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782594136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[0]R flush_numbers[0..12]: [107, 116, 111, 122, 118, 113, 122, 118, 122, 122, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782685718Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[0]R generation[0..12]: [65, 68, 66, 71, 70, 67, 71, 70, 71, 71, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782702415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782722024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[1]R flush_numbers[0..12]: [107, 116, 111, 122, 118, 113, 122, 118, 122, 122, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782738111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[1]R generation[0..12]: [65, 68, 66, 71, 70, 67, 71, 70, 71, 71, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782753378Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782769936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[2]R flush_numbers[0..12]: [107, 116, 111, 122, 118, 113, 122, 118, 122, 122, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782785793Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[2]R generation[0..12]: [65, 68, 66, 71, 70, 67, 71, 70, 71, 71, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.78280093Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782817187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:03:17.782830714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782851183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Next flush: 123","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782864599Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782877835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"":"downstairs","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782891721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782904777Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"0","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782918954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"1","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782937222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"client":"2","":"downstairs","session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782951809Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782965305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d is now active with session: d79cc885-993d-4f86-8a90-8d2ca1dcac5e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782980322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.782993047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490,"session_id":"d79cc885-993d-4f86-8a90-8d2ca1dcac5e"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.783017279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} {"msg":"Activated sub_volume c24d691f-d3b0-4bbe-b0f5-ad38b5122a3d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:17.783028874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 337 len 20480 data: 4 6 7 7 7 02/30 Read block 30 len 28672 03/30 Write block 431 len 12288 data: 10 10 9 04/30 Read block 415 len 8192 05/30 Read block 368 len 24576 06/30 Write block 404 len 40960 data: 2 2 3 6 6 7 7 8 9 8 07/30 Flush 08/30 Write block 145 len 24576 data: 4 4 4 5 5 6 09/30 Read block 165 len 24576 10/30 Write block 182 len 12288 data: 7 7 6 11/30 Write block 115 len 20480 data: 5 5 5 4 5 12/30 Write block 580 len 24576 data: 11 12 10 10 11 10 13/30 Read block 349 len 16384 14/30 Write block 475 len 4096 data: 5 15/30 Write block 101 len 8192 data: 4 3 16/30 Read block 14 len 32768 17/30 Write block 312 len 16384 data: 6 4 5 7 18/30 Write block 2 len 20480 data: 3 5 5 7 8 19/30 Read block 385 len 36864 20/30 Write block 546 len 24576 data: 7 6 8 7 9 8 21/30 Write block 463 len 8192 data: 6 8 22/30 Write block 578 len 28672 data: 8 9 12 13 11 11 12 23/30 Read block 227 len 32768 24/30 Flush 25/30 Write block 333 len 24576 data: 3 3 3 3 5 7 26/30 Flush 27/30 Write block 131 len 20480 data: 12 8 9 12 12 28/30 Read block 592 len 32768 29/30 Read block 273 len 32768 30/30 Read block 386 len 40960 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 0 26 Done 4 30 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:17.795438462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1490} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 72 72 107 124 124 F F F <--- 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 71 71 71 122 122 122 F F F 4 080-099 70 70 70 118 118 118 F F F 5 100-119 67 72 72 113 124 124 F F F <--- 6 120-139 71 71 71 122 122 122 F T T <--- 7 140-159 70 72 72 118 124 124 F F F <--- 8 160-179 71 71 71 122 122 122 F F F 9 180-199 71 72 72 122 124 124 F F F <--- 10 200-219 62 62 62 104 104 104 F F F 11 220-239 70 70 70 119 119 119 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 69 69 117 117 117 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 70 72 72 118 124 124 F F F <--- 16 320-339 60 72 72 101 125 125 T F F <--- 17 340-359 69 72 72 117 123 123 T F F <--- 18 360-379 69 69 69 117 117 117 F F F 19 380-399 70 70 70 121 121 121 F F F 20 400-419 68 72 72 115 123 123 F F F <--- 21 420-439 69 72 72 117 123 123 T F F <--- 22 440-459 63 63 63 105 105 105 F F F 23 460-479 54 72 72 89 124 124 F F F <--- 24 480-499 66 66 66 112 112 112 F F F 25 500-519 70 70 70 118 118 118 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 71 72 72 122 124 124 F F F <--- 28 560-579 67 72 72 113 124 124 F F F <--- 29 580-599 66 72 72 112 124 124 F F F <--- Max gen: 72, Max flush: 125 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 71 71 71 122 122 122 F F F 4 080-099 70 70 70 118 118 118 F F F 5 100-119 72 72 72 124 124 124 F F F 6 120-139 73 73 73 126 126 126 F F F 7 140-159 72 72 72 124 124 124 F F F 8 160-179 71 71 71 122 122 122 F F F 9 180-199 72 72 72 124 124 124 F F F 10 200-219 62 62 62 104 104 104 F F F 11 220-239 70 70 70 119 119 119 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 69 69 117 117 117 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 72 72 72 124 124 124 F F F 16 320-339 72 72 72 125 125 125 F F F 17 340-359 72 72 72 123 123 123 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 70 70 70 121 121 121 F F F 20 400-419 72 72 72 123 123 123 F F F 21 420-439 72 72 72 123 123 123 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 72 72 72 124 124 124 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 70 70 70 118 118 118 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 72 72 72 124 124 124 F F F 28 560-579 72 72 72 124 124 124 F F F 29 580-599 72 72 72 124 124 124 F F F Max gen: 73, Max flush: 126 Begin loop 37 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.136008517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:22.14662845Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.146678073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.156863358Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157146957Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157182974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15719549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157213628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"Crucible 7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109 has session id: 5d958454-2c96-45e0-8e3a-42c38a08526c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157555874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Upstairs opts: Upstairs UUID: 7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15759265Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Crucible stats registered with UUID: 7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157608397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:22.157621834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157780936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157809789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157825016Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15794373Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157959127Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157979977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.157992953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158017063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15803115Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158045006Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158058963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158072979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"ds_connection connected from Ok(127.0.0.1:35387)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158090387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"ds_connection connected from Ok(127.0.0.1:51319)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158105494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"ds_connection connected from Ok(127.0.0.1:42184)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158254171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"io task","client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15886711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158915783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.158944075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159244912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159285821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 122, 124, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159348079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[0]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 71, 72, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159365687Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159382675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 122, 124, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159398562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[1]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 71, 72, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159414189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159430547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 122, 124, 104, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159449926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[2]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 71, 72, 62, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159465233Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15948151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159500299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159514595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Next flush: 127","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159527891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159541197Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"":"downstairs","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159555073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159567439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"0","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159581075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"1","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Transition from WaitQuorum to Active","v":0Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159597523Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"client":"2","":"downstairs","session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.15961263Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159626226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109 is now active with session: 5d958454-2c96-45e0-8e3a-42c38a08526c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159641513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159654429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498,"session_id":"5d958454-2c96-45e0-8e3a-42c38a08526c"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159667835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} {"msg":"Activated sub_volume 7a9e3f1c-33d6-4a1a-a9cb-2b8754a4b109","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:22.159681361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 205 len 32768 data: 3 4 4 3 3 6 6 6 03/30 Read block 185 len 16384 04/30 Read block 62 len 4096 05/30 Write block 185 len 36864 data: 6 6 5 5 8 6 7 4 3 06/30 Read block 525 len 36864 07/30 Write block 177 len 20480 data: 7 7 6 7 7 08/30 Read block 396 len 4096 09/30 Write block 337 len 20480 data: 6 8 8 8 8 10/30 Read block 295 len 24576 11/30 Flush 12/30 Write block 261 len 12288 data: 6 7 8 13/30 Write block 220 len 4096 data: 6 14/30 Read block 581 len 32768 15/30 Flush 16/30 Write block 412 len 8192 data: 10 9 17/30 Write block 315 len 40960 data: 8 8 7 8 9 9 8 9 8 8 18/30 Read block 15 len 4096 19/30 Flush 20/30 Write block 342 len 8192 data: 5 6 21/30 Read block 348 len 8192 22/30 Read block 426 len 12288 23/30 Read block 159 len 4096 24/30 Read block 514 len 28672 25/30 Write block 385 len 36864 data: 9 10 10 10 10 8 7 7 7 26/30 Read block 296 len 32768 27/30 Read block 554 len 16384 28/30 Read block 489 len 36864 29/30 Write block 300 len 16384 data: 4 4 4 5 30/30 Read block 226 len 8192 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Sent Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:22.172458386Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1498} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 71 71 71 122 122 122 F F F 4 080-099 70 70 70 118 118 118 F F F 5 100-119 72 72 72 124 124 124 F F F 6 120-139 73 73 73 126 126 126 F F F 7 140-159 72 72 72 124 124 124 F F F 8 160-179 71 74 74 122 128 128 F F F <--- 9 180-199 72 74 74 124 128 128 F F F <--- 10 200-219 62 74 74 104 128 128 F F F <--- 11 220-239 70 74 74 119 129 129 F F F <--- 12 240-259 68 68 68 114 114 114 F F F 13 260-279 69 74 74 117 129 129 F F F <--- 14 280-299 70 70 70 118 118 118 F F F 15 300-319 72 74 74 124 130 130 F T T <--- 16 320-339 72 74 74 125 130 130 F F F <--- 17 340-359 72 74 74 123 128 128 F T T <--- 18 360-379 69 69 69 117 117 117 F F F 19 380-399 70 70 70 121 121 121 F T T <--- 20 400-419 72 74 74 123 130 130 F F F <--- 21 420-439 72 72 72 123 123 123 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 72 72 72 124 124 124 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 70 70 70 118 118 118 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 72 72 72 124 124 124 F F F 28 560-579 72 72 72 124 124 124 F F F 29 580-599 72 72 72 124 124 124 F F F Max gen: 74, Max flush: 130 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 71 71 71 122 122 122 F F F 4 080-099 70 70 70 118 118 118 F F F 5 100-119 72 72 72 124 124 124 F F F 6 120-139 73 73 73 126 126 126 F F F 7 140-159 72 72 72 124 124 124 F F F 8 160-179 74 74 74 128 128 128 F F F 9 180-199 74 74 74 128 128 128 F F F 10 200-219 74 74 74 128 128 128 F F F 11 220-239 74 74 74 129 129 129 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 75 75 75 131 131 131 F F F 16 320-339 74 74 74 130 130 130 F F F 17 340-359 75 75 75 131 131 131 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 75 75 75 131 131 131 F F F 20 400-419 74 74 74 130 130 130 F F F 21 420-439 72 72 72 123 123 123 F F F 22 440-459 63 63 63 105 105 105 F F F 23 460-479 72 72 72 124 124 124 F F F 24 480-499 66 66 66 112 112 112 F F F 25 500-519 70 70 70 118 118 118 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 72 72 72 124 124 124 F F F 28 560-579 72 72 72 124 124 124 F F F 29 580-599 72 72 72 124 124 124 F F F Max gen: 75, Max flush: 131 Begin loop 38 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.501373207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.51322145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.513544187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.513585896Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.513598722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.51361642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"Crucible 6aef3c68-325f-4073-9681-b71d040035cf has session id: bb4f1069-87de-43ae-918a-3b201b96c15c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.513932423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Upstairs opts: Upstairs UUID: 6aef3c68-325f-4073-9681-b71d040035cf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.5139678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Crucible stats registered with UUID: 6aef3c68-325f-4073-9681-b71d040035cf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.513983857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:26.514010849Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514155135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514179836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514194213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514313357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514329814Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"6aef3c68-325f-4073-9681-b71d040035cf active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514359928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514373594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514387691Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514401457Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514415293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.51442981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514444196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"ds_connection connected from Ok(127.0.0.1:58082)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514527174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"ds_connection connected from Ok(127.0.0.1:59901)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514641196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"ds_connection connected from Ok(127.0.0.1:47345)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.514667238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"io task","client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515290702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515319525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515344376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515663401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515681469Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 128, 128, 128, 129]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515739986Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[0]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515756514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515772901Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 128, 128, 128, 129]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515788768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[1]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515804185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515820903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 111, 122, 118, 124, 126, 124, 128, 128, 128, 129]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.51583668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[2]R generation[0..12]: [72, 68, 66, 71, 70, 72, 73, 72, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515852177Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515868385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515881811Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.51590267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Next flush: 132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515915726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515929192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"":"downstairs","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515943729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515958325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"0","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515972972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"1","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.515987159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"client":"2","":"downstairs","session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516004246Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516018083Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"6aef3c68-325f-4073-9681-b71d040035cf is now active with session: bb4f1069-87de-43ae-918a-3b201b96c15c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516033009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516046015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506,"session_id":"bb4f1069-87de-43ae-918a-3b201b96c15c"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516059211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} {"msg":"Activated sub_volume 6aef3c68-325f-4073-9681-b71d040035cf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:26.516070416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 354 len 24576 data: 7 7 6 6 8 6 02/30 Write block 68 len 8192 data: 8 9 03/30 Read block 547 len 40960 04/30 Write block 127 len 28672 data: 11 9 9 10 13 9 10 05/30 Write block 218 len 32768 data: 5 5 7 7 6 8 9 10 06/30 Write block 461 len 36864 data: 5 4 7 9 7 6 6 5 6 07/30 Flush 08/30 Write block 584 len 40960 data: 13 11 9 6 7 6 6 5 6 7 09/30 Write block 151 len 28672 data: 7 7 6 7 6 6 5 10/30 Read block 273 len 20480 11/30 Write block 432 len 8192 data: 11 10 12/30 Read block 265 len 28672 13/30 Write block 418 len 24576 data: 4 4 4 3 5 5 14/30 Write block 112 len 8192 data: 6 6 15/30 Write block 501 len 28672 data: 7 6 8 9 10 8 9 16/30 Write block 448 len 32768 data: 6 6 6 6 7 6 4 4 17/30 Read block 341 len 28672 18/30 Write block 187 len 40960 data: 6 6 9 7 8 5 4 6 6 6 19/30 Write block 416 len 32768 data: 6 5 5 5 5 4 6 6 20/30 Write block 149 len 40960 data: 6 7 8 8 7 8 7 7 6 7 21/30 Write block 543 len 36864 data: 8 8 8 8 7 9 8 10 9 22/30 Write block 82 len 12288 data: 5 5 6 23/30 Write block 387 len 12288 data: 11 11 11 24/30 Write block 349 len 12288 data: 8 6 8 25/30 Write block 170 len 36864 data: 7 7 7 6 6 5 6 8 8 26/30 Write block 396 len 20480 data: 6 6 6 6 6 27/30 Write block 508 len 40960 data: 8 7 9 9 10 9 10 7 6 6 28/30 Write block 499 len 32768 data: 10 10 8 7 9 10 11 9 29/30 Write block 225 len 8192 data: 11 9 30/30 Read block 546 len 12288 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 10 Done Done Done false 1008 Acked Write 7 Done Done Done false 1009 Acked Read 5 Done Done Done false 1010 Acked Write 2 Done Done Done false 1011 Acked Read 7 Done Done Done false 1012 Acked Write 6 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 18 18 Done 23 23 5 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:26.527224653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1506} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 70 70 70 118 118 118 T T F <--- 5 100-119 72 72 72 124 124 124 T T F <--- 6 120-139 76 76 76 132 132 132 F F F 7 140-159 72 72 72 124 124 124 T T T 8 160-179 74 74 74 128 128 128 T T F <--- 9 180-199 74 74 74 128 128 128 T T F <--- 10 200-219 76 76 76 132 132 132 F F F 11 220-239 76 76 76 132 132 132 T T F <--- 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 75 75 75 131 131 131 F F F 16 320-339 74 74 74 130 130 130 F F F 17 340-359 76 76 76 132 132 132 T T F <--- 18 360-379 69 69 69 117 117 117 F F F 19 380-399 75 75 75 131 131 131 T T F <--- 20 400-419 74 74 74 130 130 130 T T F <--- 21 420-439 72 72 72 123 123 123 T T T 22 440-459 63 63 63 105 105 105 T T F <--- 23 460-479 76 76 76 132 132 132 F F F 24 480-499 66 66 66 112 112 112 T T F <--- 25 500-519 70 70 70 118 118 118 T T F <--- 26 520-539 69 69 69 117 117 117 F F F 27 540-559 72 72 72 124 124 124 T T F <--- 28 560-579 72 72 72 124 124 124 F F F 29 580-599 72 72 72 124 124 124 T T T Max gen: 76, Max flush: 132 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 66 66 111 111 111 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 77 77 77 133 133 133 F F F 6 120-139 76 76 76 132 132 132 F F F 7 140-159 77 77 77 133 133 133 F F F 8 160-179 77 77 77 133 133 133 F F F 9 180-199 77 77 77 133 133 133 F F F 10 200-219 76 76 76 132 132 132 F F F 11 220-239 77 77 77 133 133 133 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 75 75 75 131 131 131 F F F 16 320-339 74 74 74 130 130 130 F F F 17 340-359 77 77 77 133 133 133 F F F 18 360-379 69 69 69 117 117 117 F F F 19 380-399 77 77 77 133 133 133 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 77 77 77 133 133 133 F F F 22 440-459 77 77 77 133 133 133 F F F 23 460-479 76 76 76 132 132 132 F F F 24 480-499 77 77 77 133 133 133 F F F 25 500-519 77 77 77 133 133 133 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 77 77 77 133 133 133 F F F 28 560-579 72 72 72 124 124 124 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 77, Max flush: 133 Begin loop 39 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.867329477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:30.877960586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.878008438Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888212822Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888505665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888542792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.8885588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888577208Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"Crucible 07040c05-df42-4583-b8bf-a5a4c298cd96 has session id: e610ee51-2c1f-4838-9a05-287a23757393","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888915292Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Upstairs opts: Upstairs UUID: 07040c05-df42-4583-b8bf-a5a4c298cd96, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.888940633Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Crucible stats registered with UUID: 07040c05-df42-4583-b8bf-a5a4c298cd96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.8889551Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:30.888968066Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889124907Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889148728Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889163765Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889302458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889317585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"07040c05-df42-4583-b8bf-a5a4c298cd96 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889338745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.88935148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889365467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889379363Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889392929Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889409697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889423823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"ds_connection connected from Ok(127.0.0.1:61157)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.88943883Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"ds_connection connected from Ok(127.0.0.1:33401)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889453527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"ds_connection connected from Ok(127.0.0.1:53249)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.889468364Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"io task","client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890282044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890320702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890337759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890658565Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890704786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 111, 132, 133, 133, 132, 133, 133, 133, 132, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890761622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[0]R generation[0..12]: [72, 68, 66, 76, 77, 77, 76, 77, 77, 77, 76, 77]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.8907784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890795178Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 111, 132, 133, 133, 132, 133, 133, 133, 132, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890810865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[1]R generation[0..12]: [72, 68, 66, 76, 77, 77, 76, 77, 77, 77, 76, 77]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890829593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890846761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 111, 132, 133, 133, 132, 133, 133, 133, 132, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890862468Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[2]R generation[0..12]: [72, 68, 66, 76, 77, 77, 76, 77, 77, 77, 76, 77]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890877485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890893792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890907449Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890927948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890941324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.89095446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"":"downstairs","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890968717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890981462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"0","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.890995279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"1","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891008925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"client":"2","":"downstairs","session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891023301Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:30.891036547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"07040c05-df42-4583-b8bf-a5a4c298cd96 is now active with session: e610ee51-2c1f-4838-9a05-287a23757393","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891054466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891067592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514,"session_id":"e610ee51-2c1f-4838-9a05-287a23757393"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891092203Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} {"msg":"Activated sub_volume 07040c05-df42-4583-b8bf-a5a4c298cd96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:30.891104118Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 562 len 12288 data: 3 4 6 02/30 Read block 479 len 12288 03/30 Write block 138 len 24576 data: 9 7 6 4 4 6 04/30 Read block 72 len 24576 05/30 Read block 298 len 28672 06/30 Write block 131 len 40960 data: 14 10 11 13 13 12 9 10 8 7 07/30 Write block 407 len 12288 data: 7 7 8 08/30 Write block 509 len 4096 data: 8 09/30 Flush 10/30 Write block 109 len 12288 data: 5 6 6 11/30 Write block 166 len 40960 data: 6 6 6 6 8 8 8 7 7 6 12/30 Write block 46 len 12288 data: 7 8 9 13/30 Read block 134 len 36864 14/30 Flush 15/30 Write block 435 len 36864 data: 7 8 8 10 10 7 5 5 5 16/30 Read block 91 len 32768 17/30 Write block 302 len 20480 data: 5 6 3 3 3 18/30 Read block 341 len 24576 19/30 Write block 184 len 8192 data: 7 7 20/30 Read block 270 len 8192 21/30 Read block 347 len 12288 22/30 Write block 369 len 8192 data: 9 9 23/30 Write block 332 len 40960 data: 4 4 4 4 4 7 9 9 9 9 24/30 Write block 191 len 24576 data: 9 6 5 7 7 7 25/30 Write block 226 len 36864 data: 10 10 7 8 10 11 9 7 8 26/30 Write block 513 len 8192 data: 10 11 27/30 Read block 202 len 8192 28/30 Read block 251 len 40960 29/30 Read block 230 len 16384 30/30 Write block 43 len 24576 data: 5 6 6 8 9 10 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 3 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:30.903118076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1514} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 66 78 78 111 135 135 F T T <--- 3 060-079 76 76 76 132 132 132 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 77 78 78 133 135 135 F F F <--- 6 120-139 76 78 78 132 134 134 F F F <--- 7 140-159 77 78 78 133 134 134 F F F <--- 8 160-179 77 78 78 133 135 135 F F F <--- 9 180-199 77 77 77 133 133 133 F T T <--- 10 200-219 76 76 76 132 132 132 F F F 11 220-239 77 77 77 133 133 133 F T T <--- 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 75 75 75 131 131 131 F T T <--- 16 320-339 74 74 74 130 130 130 F T T <--- 17 340-359 77 77 77 133 133 133 F T T <--- 18 360-379 69 69 69 117 117 117 F T T <--- 19 380-399 77 77 77 133 133 133 F F F 20 400-419 77 78 78 133 134 134 F F F <--- 21 420-439 77 77 77 133 133 133 F T T <--- 22 440-459 77 77 77 133 133 133 F T T <--- 23 460-479 76 76 76 132 132 132 F F F 24 480-499 77 77 77 133 133 133 F F F 25 500-519 77 78 78 133 134 134 F T T <--- 26 520-539 69 69 69 117 117 117 F F F 27 540-559 77 77 77 133 133 133 F F F 28 560-579 72 78 78 124 134 134 T F F <--- 29 580-599 77 77 77 133 133 133 F F F Max gen: 78, Max flush: 135 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 79 79 79 136 136 136 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 78 78 78 135 135 135 F F F 6 120-139 78 78 78 134 134 134 F F F 7 140-159 78 78 78 134 134 134 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 79 79 79 136 136 136 F F F 10 200-219 76 76 76 132 132 132 F F F 11 220-239 79 79 79 136 136 136 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 70 70 118 118 118 F F F 15 300-319 79 79 79 136 136 136 F F F 16 320-339 79 79 79 136 136 136 F F F 17 340-359 79 79 79 136 136 136 F F F 18 360-379 79 79 79 136 136 136 F F F 19 380-399 77 77 77 133 133 133 F F F 20 400-419 78 78 78 134 134 134 F F F 21 420-439 79 79 79 136 136 136 F F F 22 440-459 79 79 79 136 136 136 F F F 23 460-479 76 76 76 132 132 132 F F F 24 480-499 77 77 77 133 133 133 F F F 25 500-519 79 79 79 136 136 136 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 77 77 77 133 133 133 F F F 28 560-579 78 78 78 134 134 134 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 79, Max flush: 136 Begin loop 40 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.249805617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:35.260453723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.260510329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27072906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271019282Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27105856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271070676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271088514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"Crucible 83cf9e7b-bddf-408e-95ac-6e0316102351 has session id: 8d7671f3-afd1-48d9-8b86-b4ac96c012ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271429059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Upstairs opts: Upstairs UUID: 83cf9e7b-bddf-408e-95ac-6e0316102351, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271455461Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Crucible stats registered with UUID: 83cf9e7b-bddf-408e-95ac-6e0316102351","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271472809Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:35.271490046Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27167328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27169671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271711207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271829391Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271845778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"83cf9e7b-bddf-408e-95ac-6e0316102351 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271863526Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271876162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271889938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271903895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271917461Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271931697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271945874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:54516)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.271978749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:63989)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.272014645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:49533)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.272043328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"io task","client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.272690663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.272719976Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.272760475Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273099829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273136736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 136, 132, 133, 135, 134, 134, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273217843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[0]R generation[0..12]: [72, 68, 79, 76, 77, 78, 78, 78, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273235021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273251458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 136, 132, 133, 135, 134, 134, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273267225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[1]R generation[0..12]: [72, 68, 79, 76, 77, 78, 78, 78, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273282652Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27329908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 136, 132, 133, 135, 134, 134, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273314897Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[2]R generation[0..12]: [72, 68, 79, 76, 77, 78, 78, 78, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273329914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273349313Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273362959Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273377115ZWait for a query_work_queue command to finish before sending IO ","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Next flush: 137","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273396924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.27341031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"":"downstairs","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273424127Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273437333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"0","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273451509Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"1","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273465145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"client":"2","":"downstairs","session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 level":30,"time":"2026-03-14T05:03:35.273481893Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273495309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"83cf9e7b-bddf-408e-95ac-6e0316102351 is now active with session: 8d7671f3-afd1-48d9-8b86-b4ac96c012ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273510476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273522972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522,"session_id":"8d7671f3-afd1-48d9-8b86-b4ac96c012ad"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273536738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} {"msg":"Activated sub_volume 83cf9e7b-bddf-408e-95ac-6e0316102351","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:35.273548243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 544 len 28672 data: 9 9 9 8 10 9 11 02/30 Read block 383 len 36864 03/30 Read block 458 len 40960 04/30 Write block 316 len 24576 data: 9 8 9 10 10 9 05/30 Write block 561 len 4096 data: 4 06/30 Write block 152 len 8192 data: 9 8 07/30 Write block 107 len 28672 data: 4 7 6 7 7 7 7 08/30 Read block 518 len 16384 09/30 Read block 480 len 40960 10/30 Write block 299 len 16384 data: 4 5 5 6 11/30 Write block 51 len 32768 data: 8 6 9 9 9 7 6 7 12/30 Read block 49 len 4096 13/30 Read block 131 len 8192 14/30 Flush 15/30 Flush 16/30 Write block 495 len 36864 data: 7 7 8 10 11 11 9 8 10 17/30 Write block 144 len 28672 data: 6 5 5 5 6 7 8 18/30 Read block 290 len 36864 19/30 Read block 275 len 32768 20/30 Write block 317 len 40960 data: 9 10 11 11 10 10 9 9 8 10 21/30 Read block 130 len 12288 22/30 Read block 178 len 32768 23/30 Read block 449 len 4096 24/30 Write block 460 len 4096 data: 3 25/30 Read block 40 len 20480 26/30 Read block 436 len 16384 27/30 Read block 224 len 32768 28/30 Write block 105 len 16384 data: 2 4 5 8 29/30 Read block 333 len 36864 30/30 Read block 472 len 8192 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:35.28577878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1522} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 79 80 80 136 137 137 F F F <--- 3 060-079 76 76 76 132 132 132 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 78 80 80 135 137 137 F T T <--- 6 120-139 78 78 78 134 134 134 F F F 7 140-159 78 80 80 134 137 137 F T T <--- 8 160-179 78 78 78 135 135 135 F F F 9 180-199 79 79 79 136 136 136 F F F 10 200-219 76 76 76 132 132 132 F F F 11 220-239 79 79 79 136 136 136 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 70 80 80 118 137 137 F F F <--- 15 300-319 79 80 80 136 137 137 F T T <--- 16 320-339 79 80 80 136 137 137 F T T <--- 17 340-359 79 79 79 136 136 136 F F F 18 360-379 79 79 79 136 136 136 F F F 19 380-399 77 77 77 133 133 133 F F F 20 400-419 78 78 78 134 134 134 F F F 21 420-439 79 79 79 136 136 136 F F F 22 440-459 79 79 79 136 136 136 F F F 23 460-479 76 76 76 132 132 132 F T T <--- 24 480-499 77 77 77 133 133 133 F T T <--- 25 500-519 79 79 79 136 136 136 F T T <--- 26 520-539 69 69 69 117 117 117 F F F 27 540-559 77 80 80 133 137 137 T F F <--- 28 560-579 78 80 80 134 137 137 F F F <--- 29 580-599 77 77 77 133 133 133 F F F Max gen: 80, Max flush: 137 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 80 80 80 137 137 137 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 81 81 81 138 138 138 F F F 6 120-139 78 78 78 134 134 134 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 79 79 79 136 136 136 F F F 10 200-219 76 76 76 132 132 132 F F F 11 220-239 79 79 79 136 136 136 F F F 12 240-259 68 68 68 114 114 114 F F F 13 260-279 74 74 74 129 129 129 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 81 81 81 138 138 138 F F F 16 320-339 81 81 81 138 138 138 F F F 17 340-359 79 79 79 136 136 136 F F F 18 360-379 79 79 79 136 136 136 F F F 19 380-399 77 77 77 133 133 133 F F F 20 400-419 78 78 78 134 134 134 F F F 21 420-439 79 79 79 136 136 136 F F F 22 440-459 79 79 79 136 136 136 F F F 23 460-479 81 81 81 138 138 138 F F F 24 480-499 81 81 81 138 138 138 F F F 25 500-519 81 81 81 138 138 138 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 81, Max flush: 138 Begin loop 41 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.620025278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.631841476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632158701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.63220092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632213996Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632232154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"Crucible 577cef83-da04-4ec6-8c92-ba1940ecfa79 has session id: 2a7d6188-f3bb-42d5-abb6-ee5c80676293","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632623172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Upstairs opts: Upstairs UUID: 577cef83-da04-4ec6-8c92-ba1940ecfa79, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632659399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Crucible stats registered with UUID: 577cef83-da04-4ec6-8c92-ba1940ecfa79","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632674816Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:39.632687892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632847434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632872106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632886832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.632987208Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633010509Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"577cef83-da04-4ec6-8c92-ba1940ecfa79 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633023675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633039612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633055309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633069245Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633083342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633097458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.63312345Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"ds_connection connected from Ok(127.0.0.1:50524)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633143619Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"ds_connection connected from Ok(127.0.0.1:61013)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633158546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"ds_connection connected from Ok(127.0.0.1:45831)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633173213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"io task","client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.63393673Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633973357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.633991005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634328008Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634352729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 137, 132, 133, 138, 134, 138, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634414808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[0]R generation[0..12]: [72, 68, 80, 76, 77, 81, 78, 81, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634435427Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634452095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 137, 132, 133, 138, 134, 138, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634467992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[1]R generation[0..12]: [72, 68, 80, 76, 77, 81, 78, 81, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634483109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634499456Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 137, 132, 133, 138, 134, 138, 135, 136, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634515213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[2]R generation[0..12]: [72, 68, 80, 76, 77, 81, 78, 81, 78, 79, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.63453059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634546898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634560554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {Wait for a query_work_queue command to finish before sending IO "msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634580243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Next flush: 139","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634593769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634606995Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"":"downstairs","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634621152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634633797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"0","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634649595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"1","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634663381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"client":"2","":"downstairs","session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634677537Z","hostnameDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634693855Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"577cef83-da04-4ec6-8c92-ba1940ecfa79 is now active with session: 2a7d6188-f3bb-42d5-abb6-ee5c80676293","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634708982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634721597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530,"session_id":"2a7d6188-f3bb-42d5-abb6-ee5c80676293"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634735023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} {"msg":"Activated sub_volume 577cef83-da04-4ec6-8c92-ba1940ecfa79","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:39.634746829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 330 len 32768 data: 6 4 5 5 5 5 5 8 02/30 Read block 226 len 40960 03/30 Read block 234 len 36864 04/30 Write block 449 len 8192 data: 7 7 05/30 Write block 254 len 24576 data: 11 11 9 7 5 4 06/30 Flush 07/30 Write block 184 len 16384 data: 8 8 7 7 08/30 Read block 464 len 32768 09/30 Write block 274 len 20480 data: 9 9 9 9 8 10/30 Flush 11/30 Write block 343 len 40960 data: 7 6 7 7 6 8 9 7 9 7 12/30 Write block 417 len 32768 data: 6 6 6 6 5 7 7 7 13/30 Read block 426 len 16384 14/30 Read block 392 len 8192 15/30 Read block 432 len 36864 16/30 Write block 95 len 40960 data: 6 5 4 5 5 4 5 4 2 2 17/30 Read block 587 len 28672 18/30 Write block 120 len 40960 data: 5 6 8 7 9 9 10 12 10 10 19/30 Write block 363 len 32768 data: 5 7 7 5 6 8 10 10 20/30 Write block 397 len 28672 data: 7 7 7 7 6 4 2 21/30 Write block 406 len 36864 data: 4 8 8 9 8 9 11 10 6 22/30 Read block 543 len 36864 23/30 Read block 174 len 32768 24/30 Flush 25/30 Write block 258 len 20480 data: 6 5 5 7 8 26/30 Write block 489 len 40960 data: 6 6 6 6 8 10 8 8 9 11 27/30 Read block 540 len 12288 28/30 Write block 485 len 8192 data: 5 4 29/30 Write block 436 len 16384 data: 9 9 11 11 30/30 Read block 147 len 32768 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 6 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 26 27 Done 29 30 4 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:39.647990787Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1530} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 80 80 80 137 137 137 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 82 82 77 141 141 133 F F F <--- 5 100-119 82 82 81 141 141 138 F F F <--- 6 120-139 82 82 78 141 141 134 F F F <--- 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 82 82 79 140 140 136 F F F <--- 10 200-219 76 76 76 132 132 132 F F F 11 220-239 79 79 79 136 136 136 F F F 12 240-259 82 82 68 139 139 114 T T F <--- 13 260-279 82 82 74 140 140 129 T T F <--- 14 280-299 80 80 80 137 137 137 F F F 15 300-319 81 81 81 138 138 138 F F F 16 320-339 82 82 81 139 139 138 F F T <--- 17 340-359 82 82 79 141 141 136 F F F <--- 18 360-379 82 82 79 141 141 136 F F F <--- 19 380-399 82 82 77 141 141 133 F F F <--- 20 400-419 82 82 78 141 141 134 F F F <--- 21 420-439 82 82 79 141 141 136 T T F <--- 22 440-459 82 82 79 139 139 136 F F T <--- 23 460-479 81 81 81 138 138 138 F F F 24 480-499 81 81 81 138 138 138 T T F <--- 25 500-519 81 81 81 138 138 138 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 82, Max flush: 141 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 80 80 80 137 137 137 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 82 82 82 141 141 141 F F F 5 100-119 82 82 82 141 141 141 F F F 6 120-139 82 82 82 141 141 141 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 82 82 82 140 140 140 F F F 10 200-219 76 76 76 132 132 132 F F F 11 220-239 79 79 79 136 136 136 F F F 12 240-259 83 83 83 142 142 142 F F F 13 260-279 83 83 83 142 142 142 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 81 81 81 138 138 138 F F F 16 320-339 82 82 82 139 139 139 F F F 17 340-359 82 82 82 141 141 141 F F F 18 360-379 82 82 82 141 141 141 F F F 19 380-399 82 82 82 141 141 141 F F F 20 400-419 82 82 82 141 141 141 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 82 82 82 139 139 139 F F F 23 460-479 81 81 81 138 138 138 F F F 24 480-499 83 83 83 142 142 142 F F F 25 500-519 81 81 81 138 138 138 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 83, Max flush: 142 Begin loop 42 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.97565867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.987357745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.98767612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.987715648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.987727773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.987745712Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"Crucible 44a797b2-9de5-4404-8518-97058df6a5dd has session id: bb389517-caef-43d1-9cc0-75cb38ba1f6b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988124494Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Upstairs opts: Upstairs UUID: 44a797b2-9de5-4404-8518-97058df6a5dd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988162041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Crucible stats registered with UUID: 44a797b2-9de5-4404-8518-97058df6a5dd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988177928Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:43.988191114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988342843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988366424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988380951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988499004Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988515612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"44a797b2-9de5-4404-8518-97058df6a5dd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.98853348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988546306Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988560232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988573989Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988625592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988651104Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.98866622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"ds_connection connected from Ok(127.0.0.1:42613)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988681507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"ds_connection connected from Ok(127.0.0.1:53381)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.98870911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"ds_connection connected from Ok(127.0.0.1:60263)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.988827714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"io task","client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989401545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989446886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989464374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989767762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989810972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[0]R flush_numbers[0..12]: [124, 116, 137, 132, 141, 141, 141, 138, 135, 140, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989861024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[0]R generation[0..12]: [72, 68, 80, 76, 82, 82, 82, 81, 78, 82, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989877822Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989894279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[1]R flush_numbers[0..12]: [124, 116, 137, 132, 141, 141, 141, 138, 135, 140, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989909947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[1]R generation[0..12]: [72, 68, 80, 76, 82, 82, 82, 81, 78, 82, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989925444Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989942001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[2]R flush_numbers[0..12]: [124, 116, 137, 132, 141, 141, 141, 138, 135, 140, 132, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989957848Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[2]R generation[0..12]: [72, 68, 80, 76, 82, 82, 82, 81, 78, 82, 76, 79]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989976327Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.989992924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990017966Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-03-14T05:03:43.990038015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Next flush: 143","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990051471Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990064687Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"":"downstairs","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990078643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990091319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"0","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990105355Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"1","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990119412Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"client":"2","":"downstairs","session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990133248Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990146424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P",Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"44a797b2-9de5-4404-8518-97058df6a5dd is now active with session: bb389517-caef-43d1-9cc0-75cb38ba1f6b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990163882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990176568Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538,"session_id":"bb389517-caef-43d1-9cc0-75cb38ba1f6b"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.990190054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} {"msg":"Activated sub_volume 44a797b2-9de5-4404-8518-97058df6a5dd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:43.99020394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 396 len 40960 data: 7 8 8 8 8 7 5 3 3 3 03/30 Write block 505 len 24576 data: 12 10 10 9 9 10 04/30 Write block 5 len 24576 data: 8 9 6 9 9 8 05/30 Read block 383 len 8192 06/30 Write block 355 len 24576 data: 8 7 7 9 7 6 07/30 Read block 22 len 28672 08/30 Read block 38 len 24576 09/30 Write block 231 len 12288 data: 12 10 8 10/30 Read block 551 len 16384 11/30 Read block 4 len 20480 12/30 Write block 195 len 24576 data: 8 8 5 4 2 2 13/30 Write block 44 len 24576 data: 7 7 9 10 11 8 14/30 Read block 115 len 28672 15/30 Read block 97 len 12288 16/30 Read block 501 len 12288 17/30 Write block 307 len 20480 data: 5 5 6 7 8 18/30 Read block 357 len 24576 19/30 Write block 228 len 40960 data: 8 9 11 13 11 9 9 8 8 6 20/30 Read block 519 len 8192 21/30 Write block 457 len 40960 data: 5 5 4 4 6 5 8 10 8 7 22/30 Read block 41 len 36864 23/30 Write block 479 len 4096 data: 8 24/30 Read block 478 len 16384 25/30 Read block 355 len 36864 26/30 Write block 307 len 8192 data: 6 6 27/30 Write block 257 len 32768 data: 8 7 6 6 8 9 9 7 28/30 Read block 63 len 16384 29/30 Read block 372 len 16384 30/30 Write block 123 len 40960 data: 8 10 10 11 13 11 11 11 15 11 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Done Done false 1002 Acked Write 6 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:44.00288027Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1538} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 124 124 124 T T F <--- 1 020-039 68 68 68 116 116 116 F F F 2 040-059 80 80 80 137 137 137 T T F <--- 3 060-079 76 76 76 132 132 132 F F F 4 080-099 82 82 82 141 141 141 F F F 5 100-119 82 82 82 141 141 141 F F F 6 120-139 82 82 82 141 141 141 T T F <--- 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 82 82 82 140 140 140 T T F <--- 10 200-219 76 76 76 132 132 132 T T F <--- 11 220-239 79 79 79 136 136 136 T T F <--- 12 240-259 83 83 83 142 142 142 T T F <--- 13 260-279 83 83 83 142 142 142 T T F <--- 14 280-299 80 80 80 137 137 137 F F F 15 300-319 81 81 81 138 138 138 T T F <--- 16 320-339 82 82 82 139 139 139 F F F 17 340-359 82 82 82 141 141 141 T T F <--- 18 360-379 82 82 82 141 141 141 T T F <--- 19 380-399 82 82 82 141 141 141 T T T 20 400-419 82 82 82 141 141 141 T T T 21 420-439 83 83 83 142 142 142 F F F 22 440-459 82 82 82 139 139 139 T T F <--- 23 460-479 81 81 81 138 138 138 T T F <--- 24 480-499 83 83 83 142 142 142 F F F 25 500-519 81 81 81 138 138 138 T T F <--- 26 520-539 69 69 69 117 117 117 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 83, Max flush: 142 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 68 68 68 116 116 116 F F F 2 040-059 84 84 84 143 143 143 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 82 82 82 141 141 141 F F F 5 100-119 82 82 82 141 141 141 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 84 84 84 143 143 143 F F F 10 200-219 84 84 84 143 143 143 F F F 11 220-239 84 84 84 143 143 143 F F F 12 240-259 84 84 84 143 143 143 F F F 13 260-279 84 84 84 143 143 143 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 84 84 84 143 143 143 F F F 16 320-339 82 82 82 139 139 139 F F F 17 340-359 84 84 84 143 143 143 F F F 18 360-379 84 84 84 143 143 143 F F F 19 380-399 84 84 84 143 143 143 F F F 20 400-419 84 84 84 143 143 143 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 84 84 84 143 143 143 F F F 23 460-479 84 84 84 143 143 143 F F F 24 480-499 83 83 83 142 142 142 F F F 25 500-519 84 84 84 143 143 143 F F F 26 520-539 69 69 69 117 117 117 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 84, Max flush: 143 Begin loop 43 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.345512225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:48.356241308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.356298544Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.366612468Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.366908773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.366947611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.366959936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.366977714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"Crucible 407cf96e-a52c-4b95-9ab4-e1e8177d2764 has session id: c0480cee-4944-430e-874b-6b5151c4f6ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367345992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Upstairs opts: Upstairs UUID: 407cf96e-a52c-4b95-9ab4-e1e8177d2764, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367381908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Crucible stats registered with UUID: 407cf96e-a52c-4b95-9ab4-e1e8177d2764","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367397655Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:48.367411292Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.3675621Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"2","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367590693Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36760575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367728906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367744093Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"407cf96e-a52c-4b95-9ab4-e1e8177d2764 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367766513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367779139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367793055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367807072Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"2","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36784594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"2","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367871291Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367886938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:44831)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.367943714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"2","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:49316)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.368108319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:35997)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36813183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"io task","client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.368652307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"2","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.368694416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.368716796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369068086Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369125482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[0]R flush_numbers[0..12]: [143, 116, 143, 132, 141, 141, 143, 138, 135, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369193573Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[0]R generation[0..12]: [84, 68, 84, 76, 82, 82, 84, 81, 78, 84, 84, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36921028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369227018Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[1]R flush_numbers[0..12]: [143, 116, 143, 132, 141, 141, 143, 138, 135, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369242795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[1]R generation[0..12]: [84, 68, 84, 76, 82, 82, 84, 81, 78, 84, 84, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369258182Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36927449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[2]R flush_numbers[0..12]: [143, 116, 143, 132, 141, 141, 143, 138, 135, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369290327Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[2]R generation[0..12]: [84, 68, 84, 76, 82, 82, 84, 81, 78, 84, 84, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369305374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369321401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Max found gen is 85","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369335217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Generation requested: 86 >= found:85","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369355587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Next flush: 144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369371424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.36938466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"":"downstairs","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369398556Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369411332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"0","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369425578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"1","":"downstairs","session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369439265Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369456102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369469699Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"407cf96e-a52c-4b95-9ab4-e1e8177d2764 is now active with session: c0480cee-4944-430e-874b-6b5151c4f6ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369484815Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369497331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546,"session_id":"c0480cee-4944-430e-874b-6b5151c4f6ad"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369510627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} {"msg":"Activated sub_volume 407cf96e-a52c-4b95-9ab4-e1e8177d2764","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:48.369521812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 313 len 40960 data: 5 6 9 10 10 11 12 12 11 11 02/30 Write block 525 len 32768 data: 4 3 3 4 7 7 8 10 03/30 Write block 406 len 28672 data: 5 9 9 10 9 10 12 04/30 Read block 70 len 32768 05/30 Write block 449 len 28672 data: 8 8 7 8 7 5 5 06/30 Write block 397 len 24576 data: 9 9 9 9 8 6 07/30 Read block 50 len 20480 08/30 Write block 102 len 12288 data: 5 3 3 09/30 Read block 358 len 20480 10/30 Write block 368 len 32768 data: 9 11 11 8 10 10 12 10 11/30 Flush 12/30 Write block 236 len 20480 data: 9 7 5 3 3 13/30 Read block 176 len 12288 14/30 Write block 195 len 12288 data: 9 9 6 15/30 Write block 22 len 32768 data: 3 3 6 6 6 5 5 5 16/30 Read block 93 len 28672 17/30 Read block 564 len 20480 18/30 Write block 306 len 12288 data: 4 7 7 19/30 Write block 49 len 12288 data: 9 8 9 20/30 Read block 361 len 28672 21/30 Read block 520 len 4096 22/30 Write block 97 len 8192 data: 5 6 23/30 Write block 230 len 24576 data: 12 14 12 10 10 9 24/30 Read block 130 len 40960 25/30 Write block 197 len 4096 data: 7 26/30 Read block 572 len 24576 27/30 Read block 419 len 16384 28/30 Read block 533 len 12288 29/30 Read block 285 len 12288 30/30 Write block 316 len 36864 data: 11 11 12 13 13 12 12 10 10 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 5 Done Done Done false 1012 Acked Read 3 Done Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 1 1 19 Done 2 18 18 38 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:48.381675874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1546} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 68 68 68 116 116 116 F T T <--- 2 040-059 84 84 84 143 143 143 F T T <--- 3 060-079 76 76 76 132 132 132 F F F 4 080-099 82 82 82 141 141 141 F T T <--- 5 100-119 86 86 86 144 144 144 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 84 84 84 143 143 143 F T T <--- 10 200-219 84 84 84 143 143 143 F F F 11 220-239 84 84 84 143 143 143 T T T 12 240-259 84 84 84 143 143 143 T T T 13 260-279 84 84 84 143 143 143 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 86 86 86 144 144 144 F T T <--- 16 320-339 86 86 86 144 144 144 F T T <--- 17 340-359 84 84 84 143 143 143 F F F 18 360-379 86 86 86 144 144 144 F F F 19 380-399 86 86 86 144 144 144 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 86 86 86 144 144 144 F F F 23 460-479 84 84 84 143 143 143 F F F 24 480-499 83 83 83 142 142 142 F F F 25 500-519 84 84 84 143 143 143 F F F 26 520-539 86 86 86 144 144 144 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 86, Max flush: 144 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 233,235c233,237 < 11, < 13, < 11, --- > 12, > 14, > 12, > 10, > 10, 238,239d239 < 8, < 9, diff found after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 87 87 87 145 145 145 F F F 5 100-119 86 86 86 144 144 144 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 87 87 87 145 145 145 F F F 10 200-219 84 84 84 143 143 143 F F F 11 220-239 87 87 87 145 145 145 F F F 12 240-259 87 87 87 145 145 145 F F F 13 260-279 84 84 84 143 143 143 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 87 87 87 145 145 145 F F F 17 340-359 84 84 84 143 143 143 F F F 18 360-379 86 86 86 144 144 144 F F F 19 380-399 86 86 86 144 144 144 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 86 86 86 144 144 144 F F F 23 460-479 84 84 84 143 143 143 F F F 24 480-499 83 83 83 142 142 142 F F F 25 500-519 84 84 84 143 143 143 F F F 26 520-539 86 86 86 144 144 144 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 87, Max flush: 145 Begin loop 44 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.723501251Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735161018Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735475021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735512388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735524744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735546073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"Crucible 0c1d48f8-05d4-4f91-8511-abd3eac22f3b has session id: 03095bbd-da01-4208-9177-97ceb32e1713","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735908748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Upstairs opts: Upstairs UUID: 0c1d48f8-05d4-4f91-8511-abd3eac22f3b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735943404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Crucible stats registered with UUID: 0c1d48f8-05d4-4f91-8511-abd3eac22f3b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.735958001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:52.735970837Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736162674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.73619735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736212947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736315584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.73633006Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"0c1d48f8-05d4-4f91-8511-abd3eac22f3b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736349489Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736362545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736376651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736390738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736404364Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736421312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736435898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"ds_connection connected from Ok(127.0.0.1:50667)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736476597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"ds_connection connected from Ok(127.0.0.1:57812)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736492964Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"ds_connection connected from Ok(127.0.0.1:46264)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.736508241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737283094Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737322532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737338909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737652172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737695482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 145, 132, 145, 144, 143, 138, 135, 145, 143, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737754399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[0]R generation[0..12]: [84, 87, 87, 76, 87, 86, 84, 81, 78, 87, 84, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737771136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737787674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 145, 132, 145, 144, 143, 138, 135, 145, 143, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737803691Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[1]R generation[0..12]: [84, 87, 87, 76, 87, 86, 84, 81, 78, 87, 84, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737819098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737838557Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 145, 132, 145, 144, 143, 138, 135, 145, 143, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737854324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[2]R generation[0..12]: [84, 87, 87, 76, 87, 86, 84, 81, 78, 87, 84, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737869501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737885778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Max found gen is 88","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:03:52.737905767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737919844Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Next flush: 146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.73793306Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737946476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"":"downstairs","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737960772Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737973618Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"0","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.737987745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"1","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:03:52.738002041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"client":"2","":"downstairs","session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738027563Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738040749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"0c1d48f8-05d4-4f91-8511-abd3eac22f3b is now active with session: 03095bbd-da01-4208-9177-97ceb32e1713","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738055586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738069832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557,"session_id":"03095bbd-da01-4208-9177-97ceb32e1713"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738083288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} {"msg":"Activated sub_volume 0c1d48f8-05d4-4f91-8511-abd3eac22f3b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:52.738094903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 528 len 8192 data: 5 8 02/30 Read block 346 len 12288 03/30 Write block 335 len 8192 data: 6 6 04/30 Write block 103 len 8192 data: 4 4 05/30 Write block 254 len 4096 data: 12 06/30 Write block 226 len 36864 data: 11 11 9 10 12 14 12 10 10 07/30 Write block 352 len 12288 data: 8 7 8 08/30 Flush 09/30 Write block 189 len 28672 data: 10 8 10 7 6 8 10 10/30 Flush 11/30 Write block 329 len 40960 data: 8 7 5 6 6 6 7 7 9 10 12/30 Write block 523 len 24576 data: 4 4 5 4 4 6 13/30 Read block 186 len 4096 14/30 Write block 80 len 4096 data: 10 15/30 Read block 276 len 4096 16/30 Flush 17/30 Write block 386 len 36864 data: 11 12 12 12 9 8 8 8 8 18/30 Write block 486 len 20480 data: 5 4 4 7 7 19/30 Write block 377 len 32768 data: 9 8 10 11 8 11 12 12 20/30 Read block 200 len 4096 21/30 Flush 22/30 Write block 378 len 32768 data: 9 11 12 9 12 13 13 10 23/30 Write block 489 len 24576 data: 8 8 7 7 9 11 24/30 Write block 563 len 32768 data: 5 7 7 8 7 9 12 10 25/30 Read block 533 len 32768 26/30 Read block 371 len 32768 27/30 Read block 202 len 24576 28/30 Read block 513 len 40960 29/30 Read block 32 len 24576 30/30 Write block 323 len 4096 data: 11 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 7 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 22 1 24 Done 21 0 21 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1007)) Some(JobId(1020)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:52.749392325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1557} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 88 87 88 148 145 148 F F F <--- 5 100-119 88 88 88 146 146 146 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 88 87 88 147 145 147 F F F <--- 10 200-219 84 84 84 143 143 143 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 88 88 88 146 146 146 F F F 13 260-279 84 84 84 143 143 143 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 88 88 88 148 146 148 T F T <--- 17 340-359 88 88 88 146 146 146 F F F 18 360-379 88 86 88 149 144 149 T F T <--- 19 380-399 88 86 88 149 144 149 T F T <--- 20 400-419 86 86 86 144 144 144 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 86 86 86 144 144 144 F F F 23 460-479 84 84 84 143 143 143 F F F 24 480-499 88 83 88 149 142 149 T F T <--- 25 500-519 84 84 84 143 143 143 F F F 26 520-539 88 88 88 148 146 148 F F F <--- 27 540-559 80 80 80 137 137 137 F F F 28 560-579 80 80 80 137 137 137 T F T <--- 29 580-599 77 77 77 133 133 133 F F F Max gen: 88, Max flush: 149 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 78 78 135 135 135 F F F 9 180-199 88 88 88 147 147 147 F F F 10 200-219 84 84 84 143 143 143 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 88 88 88 146 146 146 F F F 13 260-279 84 84 84 143 143 143 F F F 14 280-299 80 80 80 137 137 137 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 89 89 89 150 150 150 F F F 17 340-359 88 88 88 146 146 146 F F F 18 360-379 89 89 89 150 150 150 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 83 83 83 142 142 142 F F F 22 440-459 86 86 86 144 144 144 F F F 23 460-479 84 84 84 143 143 143 F F F 24 480-499 89 89 89 150 150 150 F F F 25 500-519 84 84 84 143 143 143 F F F 26 520-539 88 88 88 148 148 148 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 89 89 89 150 150 150 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 89, Max flush: 150 Begin loop 45 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.065601064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:57.076188583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.076241897Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.086614108Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.086901429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.086940507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.086952412Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.08697054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"Crucible 7b32a8e8-b84c-4bbd-a600-43f02efb2e54 has session id: 46a0e42b-d2fa-4f65-8416-ea4af2c83eba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087319559Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Upstairs opts: Upstairs UUID: 7b32a8e8-b84c-4bbd-a600-43f02efb2e54, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087347352Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Crucible stats registered with UUID: 7b32a8e8-b84c-4bbd-a600-43f02efb2e54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087362259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:57.087375345Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087542331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087570193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.08758525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087696211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087710477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"7b32a8e8-b84c-4bbd-a600-43f02efb2e54 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087741311Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087755087Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087768874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.08778279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087796266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087810443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087825109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"ds_connection connected from Ok(127.0.0.1:33392)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087873101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"ds_connection connected from Ok(127.0.0.1:41362)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087889619Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"ds_connection connected from Ok(127.0.0.1:53790)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.087907977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.088663181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.088691163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.088711703Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089029678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089080361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 135, 147, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089153304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[0]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 78, 88, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089170061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089186639Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 135, 147, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089202486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[1]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 78, 88, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089218123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089234451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 135, 147, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089250198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[2]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 78, 88, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089265885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089282052Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089296009Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089316278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Next flush: 151","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089331995Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089345851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"":"downstairs","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089360278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089373644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"0","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089387851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"1","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089404708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"client":"2","":"downstairs","session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089418895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089432761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"7b32a8e8-b84c-4bbd-a600-43f02efb2e54 is now active with session: 46a0e42b-d2fa-4f65-8416-ea4af2c83eba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089447758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089460504Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565,"session_id":"46a0e42b-d2fa-4f65-8416-ea4af2c83eba"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.08947376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} {"msg":"Activated sub_volume 7b32a8e8-b84c-4bbd-a600-43f02efb2e54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:57.089485615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 425 len 24576 data: 8 8 8 9 9 10 02/30 Flush 03/30 Write block 324 len 36864 data: 11 9 11 9 9 9 8 6 7 04/30 Read block 488 len 28672 05/30 Read block 469 len 28672 06/30 Write block 174 len 4096 data: 8 07/30 Read block 256 len 4096 08/30 Read block 501 len 36864 09/30 Write block 479 len 20480 data: 9 8 9 8 6 10/30 Write block 336 len 24576 data: 8 10 11 10 10 10 11/30 Read block 33 len 36864 12/30 Read block 393 len 8192 13/30 Read block 568 len 28672 14/30 Read block 396 len 8192 15/30 Write block 514 len 12288 data: 12 8 7 16/30 Write block 267 len 28672 data: 8 7 8 7 8 8 9 17/30 Write block 421 len 20480 data: 6 8 8 8 9 18/30 Flush 19/30 Write block 276 len 28672 data: 10 10 9 8 6 6 7 20/30 Read block 374 len 4096 21/30 Write block 464 len 24576 data: 11 9 8 7 6 7 22/30 Write block 266 len 16384 data: 7 9 8 9 23/30 Write block 295 len 20480 data: 6 6 5 4 5 24/30 Read block 356 len 16384 25/30 Read block 203 len 24576 26/30 Read block 20 len 28672 27/30 Write block 185 len 32768 data: 9 8 8 7 11 9 11 8 28/30 Write block 440 len 12288 data: 8 6 6 29/30 Write block 476 len 32768 data: 4 5 6 10 9 10 9 7 30/30 Write block 468 len 32768 data: 7 8 8 8 7 7 5 6 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:57.101024847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1565} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 78 90 90 135 152 152 F F F <--- 9 180-199 88 88 88 147 147 147 F T T <--- 10 200-219 84 84 84 143 143 143 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 88 88 88 146 146 146 F F F 13 260-279 84 90 90 143 152 152 F T T <--- 14 280-299 80 80 80 137 137 137 F T T <--- 15 300-319 87 87 87 145 145 145 F F F 16 320-339 89 90 90 150 152 152 F F F <--- 17 340-359 88 90 90 146 152 152 F F F <--- 18 360-379 89 89 89 150 150 150 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 83 90 90 142 152 152 F F F <--- 22 440-459 86 86 86 144 144 144 F T T <--- 23 460-479 84 90 90 143 152 152 F T T <--- 24 480-499 89 90 90 150 152 152 F T T <--- 25 500-519 84 90 90 143 152 152 F F F <--- 26 520-539 88 88 88 148 148 148 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 89 89 89 150 150 150 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 90, Max flush: 152 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 F F F 3 060-079 76 76 76 132 132 132 F F F 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 84 84 84 143 143 143 F F F 7 140-159 81 81 81 138 138 138 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 91 91 91 153 153 153 F F F 10 200-219 84 84 84 143 143 143 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 88 88 88 146 146 146 F F F 13 260-279 91 91 91 153 153 153 F F F 14 280-299 91 91 91 153 153 153 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 90 90 90 152 152 152 F F F 18 360-379 89 89 89 150 150 150 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 91 91 91 153 153 153 F F F 23 460-479 91 91 91 153 153 153 F F F 24 480-499 91 91 91 153 153 153 F F F 25 500-519 90 90 90 152 152 152 F F F 26 520-539 88 88 88 148 148 148 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 89 89 89 150 150 150 F F F 29 580-599 77 77 77 133 133 133 F F F Max gen: 91, Max flush: 153 Begin loop 46 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.431686633Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.443522761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.443847488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.443890298Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.443907155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.443925194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"Crucible a13b0e93-0451-49f0-bfd3-87aca2b679e7 has session id: 522d4da1-21ee-4f74-8a2d-94c46aebbd52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.44424772Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Upstairs opts: Upstairs UUID: a13b0e93-0451-49f0-bfd3-87aca2b679e7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444274213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Crucible stats registered with UUID: a13b0e93-0451-49f0-bfd3-87aca2b679e7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444288829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:01.444301965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444475604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444499375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444513992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444629514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444644551Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"a13b0e93-0451-49f0-bfd3-87aca2b679e7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444660728Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444673394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444687431Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444703348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444738564Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444753331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444767237Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"ds_connection connected from Ok(127.0.0.1:62496)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.44486295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"ds_connection connected from Ok(127.0.0.1:57221)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.444997262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"ds_connection connected from Ok(127.0.0.1:52091)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445033348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445568882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445597525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445614583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445930406Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.445975967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 152, 153, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446046429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[0]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 90, 91, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446063267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446079904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 152, 153, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446099683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[1]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 90, 91, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.44611513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446131698Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 145, 132, 148, 146, 143, 138, 152, 153, 143, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446147425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[2]R generation[0..12]: [84, 87, 87, 76, 88, 88, 84, 81, 90, 91, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446162542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446178899Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446192225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,Wait for a query_work_queue command to finish before sending IO "":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446213095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Next flush: 154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446226301Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446239597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"":"downstairs","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446253914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446266749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"0","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446280726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"1","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446294852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"client":"2","":"downstairs","session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446313731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446327537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"a13b0e93-0451-49f0-bfd3-87aca2b679e7 is now active with session: 522d4da1-21ee-4f74-8a2d-94c46aebbd52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446342454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.44635507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573,"session_id":"522d4da1-21ee-4f74-8a2d-94c46aebbd52"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446368966Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} {"msg":"Activated sub_volume a13b0e93-0451-49f0-bfd3-87aca2b679e7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:01.446380821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 358 len 32768 data: 10 8 7 5 6 6 8 8 02/30 Write block 69 len 36864 data: 10 8 7 7 8 8 10 9 10 03/30 Read block 95 len 36864 04/30 Flush 05/30 Write block 591 len 16384 data: 6 7 8 8 06/30 Flush 07/30 Write block 487 len 20480 data: 5 5 9 9 8 08/30 Read block 443 len 24576 09/30 Write block 514 len 24576 data: 13 9 8 7 5 4 10/30 Read block 117 len 16384 11/30 Read block 551 len 36864 12/30 Read block 493 len 36864 13/30 Read block 499 len 32768 14/30 Read block 318 len 24576 15/30 Read block 527 len 40960 16/30 Read block 444 len 24576 17/30 Write block 125 len 16384 data: 11 12 14 12 18/30 Write block 208 len 32768 data: 4 4 7 7 7 6 6 6 19/30 Read block 319 len 12288 20/30 Read block 301 len 16384 21/30 Read block 187 len 16384 22/30 Write block 575 len 24576 data: 9 7 9 9 10 13 23/30 Read block 199 len 36864 24/30 Write block 471 len 32768 data: 9 8 8 6 7 5 6 7 25/30 Write block 139 len 32768 data: 9 8 5 5 7 7 6 6 26/30 Flush 27/30 Write block 245 len 12288 data: 8 11 12 28/30 Write block 282 len 40960 data: 8 6 6 7 8 9 9 12 11 11 29/30 Read block 62 len 28672 30/30 Write block 58 len 12288 data: 8 8 8 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:01.459304884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1573} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 87 87 87 145 145 145 T F T <--- 3 060-079 92 76 92 154 132 154 T T T <--- 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 92 84 92 156 143 156 F F F <--- 7 140-159 92 81 92 156 138 156 F F F <--- 8 160-179 90 90 90 152 152 152 F F F 9 180-199 91 91 91 153 153 153 F F F 10 200-219 92 84 92 156 143 156 F F F <--- 11 220-239 88 88 88 146 146 146 F F F 12 240-259 88 88 88 146 146 146 T F T <--- 13 260-279 91 91 91 153 153 153 F F F 14 280-299 91 91 91 153 153 153 T F T <--- 15 300-319 87 87 87 145 145 145 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 92 90 92 154 152 154 F T F <--- 18 360-379 92 89 92 154 150 154 F T F <--- 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 91 91 91 153 153 153 F F F 23 460-479 92 91 92 156 153 156 F F F <--- 24 480-499 92 91 92 156 153 156 F F F <--- 25 500-519 92 90 92 156 152 156 F F F <--- 26 520-539 88 88 88 148 148 148 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 92 89 92 156 150 156 F F F <--- 29 580-599 92 77 92 156 133 156 F F F <--- Max gen: 92, Max flush: 156 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 93 93 93 157 157 157 F F F 3 060-079 93 93 93 157 157 157 F F F 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 91 91 91 153 153 153 F F F 10 200-219 92 92 92 156 156 156 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 91 91 91 153 153 153 F F F 14 280-299 93 93 93 157 157 157 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 92 92 92 154 154 154 F F F 18 360-379 92 92 92 154 154 154 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 91 91 91 153 153 153 F F F 23 460-479 92 92 92 156 156 156 F F F 24 480-499 92 92 92 156 156 156 F F F 25 500-519 92 92 92 156 156 156 F F F 26 520-539 88 88 88 148 148 148 F F F 27 540-559 80 80 80 137 137 137 F F F 28 560-579 92 92 92 156 156 156 F F F 29 580-599 92 92 92 156 156 156 F F F Max gen: 93, Max flush: 157 Begin loop 47 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.787494504Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:05.79812145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.798175045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.808550137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.808850203Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.808889751Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.808901737Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.808919515Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"Crucible 8e33ec40-9ded-4ae4-ad14-2541dcd16f44 has session id: 357b2b02-a10b-4a21-9094-b3b2945dc31a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809302359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Upstairs opts: Upstairs UUID: 8e33ec40-9ded-4ae4-ad14-2541dcd16f44, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809339396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Crucible stats registered with UUID: 8e33ec40-9ded-4ae4-ad14-2541dcd16f44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809358285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:05.809371851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809519058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809554454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809569881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809658071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809673368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"8e33ec40-9ded-4ae4-ad14-2541dcd16f44 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809692967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809705633Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809719589Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809733305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809746751Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809760858Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809774954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"ds_connection connected from Ok(127.0.0.1:57058)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809835952Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38083)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809852519Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"ds_connection connected from Ok(127.0.0.1:34287)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.809946052Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.810603491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.810652804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.810692592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.810990107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811029385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 157, 157, 148, 146, 156, 156, 152, 153, 156, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811092804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[0]R generation[0..12]: [84, 87, 93, 93, 88, 88, 92, 92, 90, 91, 92, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811109361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811126039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 157, 157, 148, 146, 156, 156, 152, 153, 156, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811142016Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[1]R generation[0..12]: [84, 87, 93, 93, 88, 88, 92, 92, 90, 91, 92, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811157143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811173601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 157, 157, 148, 146, 156, 156, 152, 153, 156, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811189328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[2]R generation[0..12]: [84, 87, 93, 93, 88, 88, 92, 92, 90, 91, 92, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811204855Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811224924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"Wait for a query_work_queue command to finish before sending IO msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811244693Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Next flush: 158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811272686Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811286162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"":"downstairs","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811300619Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811313304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"0","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811327421Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"1","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811343688Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"client":"2","":"downstairs","session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811357835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811371071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"8e33ec40-9ded-4ae4-ad14-2541dcd16f44 is now active with session: 357b2b02-a10b-4a21-9094-b3b2945dc31a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811385937Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811398523Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581,"session_id":"357b2b02-a10b-4a21-9094-b3b2945dc31a"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811411779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} {"msg":"Activated sub_volume 8e33ec40-9ded-4ae4-ad14-2541dcd16f44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:05.811423124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 199 len 40960 data: 3 3 2 2 2 2 4 5 5 5 02/30 Read block 49 len 8192 03/30 Read block 20 len 4096 04/30 Write block 570 len 28672 data: 11 9 8 6 9 10 8 05/30 Read block 373 len 4096 06/30 Read block 517 len 12288 07/30 Read block 228 len 36864 08/30 Read block 164 len 40960 09/30 Write block 546 len 32768 data: 10 9 11 10 12 10 8 11 10/30 Read block 271 len 40960 11/30 Read block 309 len 16384 12/30 Read block 579 len 20480 13/30 Write block 268 len 4096 data: 9 14/30 Read block 189 len 4096 15/30 Write block 61 len 36864 data: 6 6 6 7 7 8 8 9 11 16/30 Write block 520 len 28672 data: 4 3 5 5 5 6 5 17/30 Read block 384 len 32768 18/30 Flush 19/30 Write block 351 len 12288 data: 10 9 8 20/30 Write block 48 len 24576 data: 12 10 9 10 7 10 21/30 Read block 445 len 36864 22/30 Read block 491 len 28672 23/30 Read block 16 len 4096 24/30 Read block 269 len 20480 25/30 Read block 216 len 24576 26/30 Write block 349 len 32768 data: 10 8 11 10 9 9 9 8 27/30 Read block 196 len 16384 28/30 Read block 76 len 4096 29/30 Read block 234 len 4096 30/30 Read block 93 len 40960 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 2 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:05.824150927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1581} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 93 93 93 157 157 157 F T T <--- 3 060-079 93 94 94 157 158 158 F F F <--- 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 91 94 94 153 158 158 T F F <--- 10 200-219 92 94 94 156 158 158 T F F <--- 11 220-239 88 88 88 146 146 146 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 91 94 94 153 158 158 F F F <--- 14 280-299 93 93 93 157 157 157 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 92 92 92 154 154 154 F T T <--- 18 360-379 92 92 92 154 154 154 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 91 91 91 153 153 153 F F F 23 460-479 92 92 92 156 156 156 F F F 24 480-499 92 92 92 156 156 156 F F F 25 500-519 92 92 92 156 156 156 F F F 26 520-539 88 94 94 148 158 158 F F F <--- 27 540-559 80 94 94 137 158 158 F F F <--- 28 560-579 92 94 94 156 158 158 F F F <--- 29 580-599 92 92 92 156 156 156 F F F Max gen: 94, Max flush: 158 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 95 95 95 159 159 159 F F F 3 060-079 94 94 94 158 158 158 F F F 4 080-099 88 88 88 148 148 148 F F F 5 100-119 88 88 88 146 146 146 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 F F F 11 220-239 88 88 88 146 146 146 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 94 94 94 158 158 158 F F F 14 280-299 93 93 93 157 157 157 F F F 15 300-319 87 87 87 145 145 145 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 92 92 92 154 154 154 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 91 91 91 153 153 153 F F F 23 460-479 92 92 92 156 156 156 F F F 24 480-499 92 92 92 156 156 156 F F F 25 500-519 92 92 92 156 156 156 F F F 26 520-539 94 94 94 158 158 158 F F F 27 540-559 94 94 94 158 158 158 F F F 28 560-579 94 94 94 158 158 158 F F F 29 580-599 92 92 92 156 156 156 F F F Max gen: 95, Max flush: 159 Begin loop 48 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.150567011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.162437093Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.162793415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.162832983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.162845489Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.162863367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"Crucible 9bb16eca-cc9f-443e-9dd3-a270c33b08ca has session id: 65be1c42-b0d1-4e91-8f3f-096ece7f10ac","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163209635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Upstairs opts: Upstairs UUID: 9bb16eca-cc9f-443e-9dd3-a270c33b08ca, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163244411Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Crucible stats registered with UUID: 9bb16eca-cc9f-443e-9dd3-a270c33b08ca","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163260018Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:10.163273244Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163432546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163467502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163483369Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163593099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163608466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"9bb16eca-cc9f-443e-9dd3-a270c33b08ca active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163623623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163636759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163650926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163664732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163678538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163692725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163706661Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:45664)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.16381456Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:53993)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163936156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:40020)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.163963738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164531687Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164559579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164576387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164880425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164895532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 159, 158, 148, 146, 156, 156, 152, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164930278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[0]R generation[0..12]: [84, 87, 95, 94, 88, 88, 92, 92, 90, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164950107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164966815Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 159, 158, 148, 146, 156, 156, 152, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164982582Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[1]R generation[0..12]: [84, 87, 95, 94, 88, 88, 92, 92, 90, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.164997859Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165014166Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 159, 158, 148, 146, 156, 156, 152, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165029943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[2]R generation[0..12]: [84, 87, 95, 94, 88, 88, 92, 92, 90, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.16504507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165061318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165074744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:04:10.16508868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Next flush: 160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165106968Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165120365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"":"downstairs","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165134481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165147047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"0","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165163024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"1","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.16517718Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"client":"2","":"downstairs","session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:04:10.165191517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165206924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"9bb16eca-cc9f-443e-9dd3-a270c33b08ca is now active with session: 65be1c42-b0d1-4e91-8f3f-096ece7f10ac","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165221941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165234446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589,"session_id":"65be1c42-b0d1-4e91-8f3f-096ece7f10ac"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165248013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} {"msg":"Activated sub_volume 9bb16eca-cc9f-443e-9dd3-a270c33b08ca","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:10.165260028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 364 len 24576 data: 9 9 6 7 10 12 02/30 Write block 426 len 32768 data: 9 9 10 10 11 11 12 11 03/30 Write block 477 len 8192 data: 7 8 04/30 Write block 425 len 4096 data: 10 05/30 Write block 494 len 20480 data: 12 9 9 10 12 06/30 Flush 07/30 Write block 86 len 24576 data: 8 6 6 8 11 12 08/30 Read block 434 len 20480 09/30 Read block 229 len 16384 10/30 Read block 433 len 4096 11/30 Read block 476 len 4096 12/30 Read block 339 len 24576 13/30 Write block 52 len 4096 data: 8 14/30 Read block 361 len 16384 15/30 Read block 510 len 8192 16/30 Read block 46 len 24576 17/30 Write block 220 len 40960 data: 8 8 7 9 10 12 12 12 10 11 18/30 Read block 50 len 40960 19/30 Write block 445 len 40960 data: 4 3 3 7 9 9 8 9 8 6 20/30 Read block 351 len 16384 21/30 Read block 258 len 36864 22/30 Read block 316 len 24576 23/30 Flush 24/30 Write block 538 len 32768 data: 7 10 8 8 7 9 10 10 25/30 Read block 356 len 16384 26/30 Read block 575 len 24576 27/30 Write block 109 len 24576 data: 7 8 8 8 8 5 28/30 Write block 440 len 28672 data: 9 7 7 6 5 5 4 29/30 Write block 308 len 4096 data: 8 30/30 Write block 440 len 28672 data: 10 8 8 7 6 6 5 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 26 3 32 Done 27 4 27 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:10.177152281Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1589} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 96 95 96 161 159 161 F F F <--- 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 88 96 161 148 161 F F F <--- 5 100-119 88 88 88 146 146 146 T F T <--- 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 F F F 11 220-239 96 88 96 161 146 161 F F F <--- 12 240-259 93 93 93 157 157 157 F F F 13 260-279 94 94 94 158 158 158 F F F 14 280-299 93 93 93 157 157 157 F F F 15 300-319 87 87 87 145 145 145 T F T <--- 16 320-339 90 90 90 152 152 152 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 92 96 160 154 160 F T F <--- 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 96 90 96 160 152 160 F T F <--- 22 440-459 96 91 96 161 153 161 T F T <--- 23 460-479 96 92 96 160 156 160 F T F <--- 24 480-499 96 92 96 160 156 160 F F F <--- 25 500-519 92 92 92 156 156 156 F F F 26 520-539 94 94 94 158 158 158 T F T <--- 27 540-559 94 94 94 158 158 158 T F T <--- 28 560-579 94 94 94 158 158 158 F F F 29 580-599 92 92 92 156 156 156 F F F Max gen: 96, Max flush: 161 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 96 96 96 161 161 161 F F F 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 96 96 161 161 161 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 F F F 11 220-239 96 96 96 161 161 161 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 94 94 94 158 158 158 F F F 14 280-299 93 93 93 157 157 157 F F F 15 300-319 97 97 97 162 162 162 F F F 16 320-339 90 90 90 152 152 152 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 89 89 89 150 150 150 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 96 96 96 160 160 160 F F F 22 440-459 97 97 97 162 162 162 F F F 23 460-479 96 96 96 160 160 160 F F F 24 480-499 96 96 96 160 160 160 F F F 25 500-519 92 92 92 156 156 156 F F F 26 520-539 97 97 97 162 162 162 F F F 27 540-559 97 97 97 162 162 162 F F F 28 560-579 94 94 94 158 158 158 F F F 29 580-599 92 92 92 156 156 156 F F F Max gen: 97, Max flush: 162 Begin loop 49 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.504402074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:14.51502585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.515072341Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.525468442Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.525756644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.525794211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.525806866Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.525824875Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"Crucible a668e4c4-a08d-44e7-8824-8c527ae1ef0a has session id: 931e5917-2aee-419a-a3ce-454e857fc59d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526183318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Upstairs opts: Upstairs UUID: a668e4c4-a08d-44e7-8824-8c527ae1ef0a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526217663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Crucible stats registered with UUID: a668e4c4-a08d-44e7-8824-8c527ae1ef0a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526233541Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:14.526247267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526381188Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526404478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526419145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526543222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526559739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"a668e4c4-a08d-44e7-8824-8c527ae1ef0a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526576717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526589132Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526602959Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526616935Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526630511Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526644788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526661345Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"ds_connection connected from Ok(127.0.0.1:34169)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526720352Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"ds_connection connected from Ok(127.0.0.1:61989)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526748005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"ds_connection connected from Ok(127.0.0.1:44456)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.526764112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.527496906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.527535263Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.527583455Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.527876478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.527910284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 152, 158, 158, 161]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.52801253Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[0]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 90, 94, 94, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528030929Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528047766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 152, 158, 158, 161]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528063694Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[1]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 90, 94, 94, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528079331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.52809962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 152, 158, 158, 161]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528116117Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[2]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 90, 94, 94, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528130974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528153534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528167281Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528181267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Next flush: 163","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528194543Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528207899Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"":"downstairs","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528221976Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528234601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"0","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:04:14.528248748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"1","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528264675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"client":"2","":"downstairs","session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528279522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528292838Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"a668e4c4-a08d-44e7-8824-8c527ae1ef0a is now active with session: 931e5917-2aee-419a-a3ce-454e857fc59d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528307705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528322872Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597,"session_id":"931e5917-2aee-419a-a3ce-454e857fc59d"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528336288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} {"msg":"Activated sub_volume a668e4c4-a08d-44e7-8824-8c527ae1ef0a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:14.528347433Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 293 len 8192 data: 9 9 02/30 Write block 390 len 40960 data: 10 9 9 9 9 6 8 10 10 10 03/30 Write block 570 len 36864 data: 12 10 9 7 10 11 9 10 10 04/30 Read block 319 len 8192 05/30 Read block 380 len 24576 06/30 Read block 396 len 36864 07/30 Read block 433 len 32768 08/30 Write block 383 len 20480 data: 14 14 11 12 13 09/30 Read block 198 len 32768 10/30 Write block 266 len 32768 data: 8 10 10 10 8 9 9 10 11/30 Read block 248 len 16384 12/30 Read block 438 len 40960 13/30 Write block 272 len 12288 data: 10 11 10 14/30 Read block 65 len 12288 15/30 Write block 538 len 20480 data: 8 11 9 9 8 16/30 Write block 512 len 24576 data: 11 11 14 10 9 8 17/30 Read block 252 len 32768 18/30 Write block 235 len 20480 data: 9 10 8 6 4 19/30 Write block 319 len 12288 data: 14 14 13 20/30 Read block 497 len 24576 21/30 Write block 311 len 12288 data: 9 7 6 22/30 Read block 47 len 28672 23/30 Read block 323 len 24576 24/30 Read block 261 len 16384 25/30 Read block 483 len 12288 26/30 Write block 320 len 4096 data: 15 27/30 Write block 172 len 28672 data: 9 8 9 7 7 9 9 28/30 Read block 59 len 16384 29/30 Write block 269 len 16384 data: 11 9 10 11 30/30 Write block 458 len 32768 data: 6 5 5 7 6 9 12 10 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 2 33 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:14.539945732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1597} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 96 96 96 161 161 161 F F F 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 96 96 161 161 161 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 90 90 90 152 152 152 F T T <--- 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 F F F 11 220-239 96 96 96 161 161 161 F T T <--- 12 240-259 93 93 93 157 157 157 F F F 13 260-279 94 94 94 158 158 158 F T T <--- 14 280-299 93 93 93 157 157 157 T T T 15 300-319 97 97 97 162 162 162 F T T <--- 16 320-339 90 90 90 152 152 152 F T T <--- 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 89 89 89 150 150 150 F T T <--- 20 400-419 86 86 86 144 144 144 F F F 21 420-439 96 96 96 160 160 160 F F F 22 440-459 97 97 97 162 162 162 F T T <--- 23 460-479 96 96 96 160 160 160 F T T <--- 24 480-499 96 96 96 160 160 160 F F F 25 500-519 92 92 92 156 156 156 F T T <--- 26 520-539 97 97 97 162 162 162 F T T <--- 27 540-559 97 97 97 162 162 162 F T T <--- 28 560-579 94 94 94 158 158 158 F T T <--- 29 580-599 92 92 92 156 156 156 F F F Max gen: 97, Max flush: 162 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 96 96 96 161 161 161 F F F 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 96 96 161 161 161 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 92 92 92 156 156 156 F F F 7 140-159 92 92 92 156 156 156 F F F 8 160-179 98 98 98 163 163 163 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 F F F 11 220-239 98 98 98 163 163 163 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 98 98 98 163 163 163 F F F 14 280-299 98 98 98 163 163 163 F F F 15 300-319 98 98 98 163 163 163 F F F 16 320-339 98 98 98 163 163 163 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 98 98 98 163 163 163 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 96 96 96 160 160 160 F F F 22 440-459 98 98 98 163 163 163 F F F 23 460-479 98 98 98 163 163 163 F F F 24 480-499 96 96 96 160 160 160 F F F 25 500-519 98 98 98 163 163 163 F F F 26 520-539 98 98 98 163 163 163 F F F 27 540-559 98 98 98 163 163 163 F F F 28 560-579 98 98 98 163 163 163 F F F 29 580-599 92 92 92 156 156 156 F F F Max gen: 98, Max flush: 163 Begin loop 50 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.901029233Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.912786265Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913103319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913145748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913158654Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913176612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"Crucible a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4 has session id: 1a54d566-dc67-4dee-bb84-12115a366851","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.91354453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Upstairs opts: Upstairs UUID: a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913572492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Crucible stats registered with UUID: a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913586919Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:18.913599885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.91376434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913803207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913819135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913916229Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913930905Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913961279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913975046Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.913988852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914002708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914016484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914030941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914045137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"ds_connection connected from Ok(127.0.0.1:63785)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914067097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"ds_connection connected from Ok(127.0.0.1:42645)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914082454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"ds_connection connected from Ok(127.0.0.1:63690)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914097201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914855306Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914890322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.914911432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915242412Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915288834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 163, 158, 158, 163]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915340007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[0]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 98, 94, 94, 98]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915356925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915373502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 163, 158, 158, 163]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915389309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[1]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 98, 94, 94, 98]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915404606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915420914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 161, 158, 161, 162, 156, 156, 163, 158, 158, 163]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915436981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[2]R generation[0..12]: [84, 87, 96, 94, 96, 97, 92, 92, 98, 94, 94, 98]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915452558Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915469926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Max found gen is 99","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915483892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Generation requested: 100 >= found:99","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915504352Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915520189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915533745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"":"downstairs","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915547771Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915560567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"0","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915574624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"client":"1","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.91558849Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"client":"2","":"downstairs","session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915605758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915619344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4 is now active with session: 1a54d566-dc67-4dee-bb84-12115a366851","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915634891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915647597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605,"session_id":"1a54d566-dc67-4dee-bb84-12115a366851"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915660823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} {"msg":"Activated sub_volume a27422f3-d69d-4a1a-ad7c-8ebcfcb863f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:18.915672278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 144 len 28672 data: 8 7 7 6 7 8 9 02/30 Read block 389 len 20480 03/30 Flush 04/30 Write block 286 len 16384 data: 9 10 10 13 05/30 Read block 90 len 32768 06/30 Write block 597 len 4096 data: 7 07/30 Flush 08/30 Flush 09/30 Write block 50 len 28672 data: 10 11 9 11 10 10 8 10/30 Read block 282 len 8192 11/30 Read block 583 len 36864 12/30 Write block 381 len 36864 data: 10 13 15 15 12 13 14 13 13 13/30 Read block 374 len 32768 14/30 Flush 15/30 Write block 164 len 40960 data: 5 6 7 7 7 7 9 9 10 9 16/30 Read block 41 len 8192 17/30 Write block 155 len 12288 data: 8 8 7 18/30 Read block 307 len 16384 19/30 Read block 313 len 12288 20/30 Read block 489 len 36864 21/30 Write block 517 len 20480 data: 9 6 5 5 4 22/30 Write block 132 len 8192 data: 12 12 23/30 Read block 531 len 4096 24/30 Flush 25/30 Write block 436 len 28672 data: 10 10 12 12 11 9 9 26/30 Read block 570 len 24576 27/30 Read block 452 len 16384 28/30 Write block 211 len 40960 data: 8 8 7 7 7 7 7 6 6 9 29/30 Read block 250 len 36864 30/30 Write block 51 len 16384 data: 12 10 12 11 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 5 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Read 9 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:18.928778843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1605} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 100 100 96 167 167 161 T T F <--- 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 96 96 161 161 161 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 100 100 92 168 168 156 F F F <--- 7 140-159 100 100 92 168 168 156 F F T <--- 8 160-179 100 100 98 168 168 163 F F F <--- 9 180-199 94 94 94 158 158 158 F F F 10 200-219 94 94 94 158 158 158 T T F <--- 11 220-239 98 98 98 163 163 163 T T F <--- 12 240-259 93 93 93 157 157 157 F F F 13 260-279 98 98 98 163 163 163 F F F 14 280-299 100 100 98 165 165 163 F F F <--- 15 300-319 98 98 98 163 163 163 F F F 16 320-339 98 98 98 163 163 163 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 100 100 98 167 167 163 F F F <--- 20 400-419 86 86 86 144 144 144 F F F 21 420-439 96 96 96 160 160 160 T T F <--- 22 440-459 98 98 98 163 163 163 T T F <--- 23 460-479 98 98 98 163 163 163 F F F 24 480-499 96 96 96 160 160 160 F F F 25 500-519 100 100 98 168 168 163 F F F <--- 26 520-539 100 100 98 168 168 163 F F F <--- 27 540-559 98 98 98 163 163 163 F F F 28 560-579 98 98 98 163 163 163 F F F 29 580-599 100 100 92 165 165 156 F F F <--- Max gen: 100, Max flush: 168 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 101 101 101 169 169 169 F F F 3 060-079 94 94 94 158 158 158 F F F 4 080-099 96 96 96 161 161 161 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 100 100 100 168 168 168 F F F 7 140-159 100 100 100 168 168 168 F F F 8 160-179 100 100 100 168 168 168 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 101 101 101 169 169 169 F F F 11 220-239 101 101 101 169 169 169 F F F 12 240-259 93 93 93 157 157 157 F F F 13 260-279 98 98 98 163 163 163 F F F 14 280-299 100 100 100 165 165 165 F F F 15 300-319 98 98 98 163 163 163 F F F 16 320-339 98 98 98 163 163 163 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 100 100 100 167 167 167 F F F 20 400-419 86 86 86 144 144 144 F F F 21 420-439 101 101 101 169 169 169 F F F 22 440-459 101 101 101 169 169 169 F F F 23 460-479 98 98 98 163 163 163 F F F 24 480-499 96 96 96 160 160 160 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 100 100 100 168 168 168 F F F 27 540-559 98 98 98 163 163 163 F F F 28 560-579 98 98 98 163 163 163 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 101, Max flush: 169 Begin loop 51 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.257275794Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.268928227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269250924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269291843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.26930889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269333111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"Crucible 69e4be00-bea5-466e-9304-cfeda95380de has session id: b159361d-0c6f-4ebe-91f4-d121b7ff4e8c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269713114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Upstairs opts: Upstairs UUID: 69e4be00-bea5-466e-9304-cfeda95380de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.26974753Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Crucible stats registered with UUID: 69e4be00-bea5-466e-9304-cfeda95380de","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269763737Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:23.269777454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.269953664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.26998876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270004587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270108014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270125072Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"69e4be00-bea5-466e-9304-cfeda95380de active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270146562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270159447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270173254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270201096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270218364Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270232671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37234)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270304994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:58586)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270435113Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:36506)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.270463356Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271090681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271120655Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271137953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271436949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271480759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 169, 158, 161, 162, 168, 168, 168, 158, 169, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271545608Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[0]R generation[0..12]: [84, 87, 101, 94, 96, 97, 100, 100, 100, 94, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271562466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271579194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 169, 158, 161, 162, 168, 168, 168, 158, 169, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271594911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[1]R generation[0..12]: [84, 87, 101, 94, 96, 97, 100, 100, 100, 94, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271612979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271629226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 169, 158, 161, 162, 168, 168, 168, 158, 169, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271644823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[2]R generation[0..12]: [84, 87, 101, 94, 96, 97, 100, 100, 100, 94, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.27166013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271676758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271690194Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271709573Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Next flush: 170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271723099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271736365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"":"downstairs","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271750432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271762997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"0","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271776734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"1","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.27179082Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"client":"2","":"downstairs","session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:04:23.271804746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271819013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"69e4be00-bea5-466e-9304-cfeda95380de is now active with session: b159361d-0c6f-4ebe-91f4-d121b7ff4e8c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271836051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271848756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613,"session_id":"b159361d-0c6f-4ebe-91f4-d121b7ff4e8c"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271862162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} {"msg":"Activated sub_volume 69e4be00-bea5-466e-9304-cfeda95380de","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:23.271873418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 397 len 4096 data: 11 02/30 Write block 256 len 32768 data: 10 9 8 7 7 9 10 10 03/30 Write block 75 len 28672 data: 11 10 11 10 10 11 7 04/30 Read block 483 len 36864 05/30 Read block 25 len 36864 06/30 Read block 519 len 28672 07/30 Write block 133 len 4096 data: 13 08/30 Write block 415 len 12288 data: 7 7 7 09/30 Write block 240 len 20480 data: 4 3 3 5 7 10/30 Write block 471 len 32768 data: 10 9 9 7 8 6 8 9 11/30 Write block 233 len 24576 data: 11 11 10 11 9 7 12/30 Read block 545 len 28672 13/30 Write block 553 len 32768 data: 12 10 9 8 6 6 6 4 14/30 Write block 390 len 40960 data: 11 10 10 10 10 7 9 12 11 11 15/30 Read block 92 len 12288 16/30 Read block 497 len 20480 17/30 Read block 1 len 24576 18/30 Read block 525 len 40960 19/30 Read block 450 len 12288 20/30 Read block 112 len 36864 21/30 Read block 287 len 20480 22/30 Flush 23/30 Write block 559 len 32768 data: 7 5 5 4 6 8 8 9 24/30 Read block 546 len 20480 25/30 Flush 26/30 Write block 156 len 12288 data: 9 8 8 27/30 Write block 422 len 28672 data: 9 9 9 11 10 10 11 28/30 Write block 483 len 12288 data: 8 5 6 29/30 Write block 457 len 16384 data: 6 7 6 6 30/30 Read block 575 len 32768 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 30 3 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:23.284071449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1613} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 101 101 101 169 169 169 F F F 3 060-079 102 94 102 170 158 170 F T F <--- 4 080-099 102 96 102 170 161 170 F T F <--- 5 100-119 97 97 97 162 162 162 F F F 6 120-139 102 100 102 170 168 170 F F F <--- 7 140-159 100 100 100 168 168 168 T F T <--- 8 160-179 100 100 100 168 168 168 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 101 101 101 169 169 169 F F F 11 220-239 102 101 102 170 169 170 F F F <--- 12 240-259 102 93 102 170 157 170 F T F <--- 13 260-279 102 98 102 170 163 170 F T F <--- 14 280-299 100 100 100 165 165 165 F F F 15 300-319 98 98 98 163 163 163 F F F 16 320-339 98 98 98 163 163 163 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 100 102 170 167 170 F T F <--- 20 400-419 102 86 102 170 144 170 F F F <--- 21 420-439 101 101 101 169 169 169 T F T <--- 22 440-459 101 101 101 169 169 169 T F T <--- 23 460-479 102 98 102 170 163 170 T F T <--- 24 480-499 96 96 96 160 160 160 T F T <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 100 100 100 168 168 168 F F F 27 540-559 102 98 102 171 163 171 F F F <--- 28 560-579 102 98 102 171 163 171 F F F <--- 29 580-599 100 100 100 165 165 165 F F F Max gen: 102, Max flush: 171 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 F F F 2 040-059 101 101 101 169 169 169 F F F 3 060-079 102 102 102 170 170 170 F F F 4 080-099 102 102 102 170 170 170 F F F 5 100-119 97 97 97 162 162 162 F F F 6 120-139 102 102 102 170 170 170 F F F 7 140-159 103 103 103 172 172 172 F F F 8 160-179 100 100 100 168 168 168 F F F 9 180-199 94 94 94 158 158 158 F F F 10 200-219 101 101 101 169 169 169 F F F 11 220-239 102 102 102 170 170 170 F F F 12 240-259 102 102 102 170 170 170 F F F 13 260-279 102 102 102 170 170 170 F F F 14 280-299 100 100 100 165 165 165 F F F 15 300-319 98 98 98 163 163 163 F F F 16 320-339 98 98 98 163 163 163 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 102 102 170 170 170 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 103 103 103 172 172 172 F F F 22 440-459 103 103 103 172 172 172 F F F 23 460-479 103 103 103 172 172 172 F F F 24 480-499 103 103 103 172 172 172 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 100 100 100 168 168 168 F F F 27 540-559 102 102 102 171 171 171 F F F 28 560-579 102 102 102 171 171 171 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 103, Max flush: 172 Begin loop 52 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.616964221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.628687236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629022649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629059706Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629072121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.62909001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"Crucible 6c6e117d-fd30-41d1-bd9f-65257fc88b0c has session id: 664f5261-6743-4efb-8880-c0d8cd2b1ee8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629437048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Upstairs opts: Upstairs UUID: 6c6e117d-fd30-41d1-bd9f-65257fc88b0c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629471403Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Crucible stats registered with UUID: 6c6e117d-fd30-41d1-bd9f-65257fc88b0c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629487521Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:27.629501167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629664571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629689462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629703909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629812148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629830086Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"6c6e117d-fd30-41d1-bd9f-65257fc88b0c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.62986069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629874537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629888653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629902609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629916556Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629930812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629945479Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"ds_connection connected from Ok(127.0.0.1:34122)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.629985017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"ds_connection connected from Ok(127.0.0.1:55501)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.630115066Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"ds_connection connected from Ok(127.0.0.1:33804)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.630136376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.63073903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.630770645Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.630795346Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631121724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631168756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[0]R flush_numbers[0..12]: [143, 145, 169, 170, 170, 162, 170, 172, 168, 158, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631227903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[0]R generation[0..12]: [84, 87, 101, 102, 102, 97, 102, 103, 100, 94, 101, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.63124473Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631261518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[1]R flush_numbers[0..12]: [143, 145, 169, 170, 170, 162, 170, 172, 168, 158, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631277305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[1]R generation[0..12]: [84, 87, 101, 102, 102, 97, 102, 103, 100, 94, 101, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631292902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.63130911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[2]R flush_numbers[0..12]: [143, 145, 169, 170, 170, 162, 170, 172, 168, 158, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631324967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[2]R generation[0..12]: [84, 87, 101, 102, 102, 97, 102, 103, 100, 94, 101, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631341074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631357221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631370778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairsWait for a query_work_queue command to finish before sending IO ","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631393988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Next flush: 173","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631407194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.63142067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"":"downstairs","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631434857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631447613Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"0","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631461459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"1","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631475215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"client":"2","":"downstairs","session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:04:27.631489562Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631503708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"6c6e117d-fd30-41d1-bd9f-65257fc88b0c is now active with session: 664f5261-6743-4efb-8880-c0d8cd2b1ee8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631518665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631531641Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621,"session_id":"664f5261-6743-4efb-8880-c0d8cd2b1ee8"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631545087Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} {"msg":"Activated sub_volume 6c6e117d-fd30-41d1-bd9f-65257fc88b0c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:27.631556652Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 270 len 24576 data: 10 11 12 12 11 10 02/30 Write block 108 len 20480 data: 9 8 9 9 9 03/30 Write block 199 len 8192 data: 4 4 04/30 Read block 563 len 16384 05/30 Write block 149 len 12288 data: 9 10 9 06/30 Write block 293 len 8192 data: 10 10 07/30 Read block 154 len 20480 08/30 Read block 568 len 20480 09/30 Read block 250 len 36864 10/30 Read block 322 len 20480 11/30 Flush 12/30 Write block 319 len 20480 data: 15 16 14 13 12 13/30 Write block 115 len 28672 data: 6 6 6 5 6 6 7 14/30 Write block 221 len 16384 data: 9 8 10 11 15/30 Write block 141 len 8192 data: 6 6 16/30 Read block 373 len 36864 17/30 Write block 34 len 12288 data: 8 7 7 18/30 Read block 524 len 32768 19/30 Read block 183 len 36864 20/30 Write block 126 len 32768 data: 13 15 13 12 12 16 13 14 21/30 Write block 232 len 8192 data: 13 12 22/30 Read block 486 len 12288 23/30 Write block 275 len 36864 data: 11 11 11 10 9 7 7 9 7 24/30 Read block 122 len 20480 25/30 Write block 314 len 20480 data: 7 10 12 12 13 26/30 Read block 216 len 16384 27/30 Write block 316 len 12288 data: 13 13 14 28/30 Write block 312 len 36864 data: 8 7 8 11 14 14 15 16 17 29/30 Write block 572 len 16384 data: 10 8 11 12 30/30 Read block 547 len 32768 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 2 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:27.642904357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1621} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 87 87 87 145 145 145 T T F <--- 2 040-059 101 101 101 169 169 169 F F F 3 060-079 102 102 102 170 170 170 F F F 4 080-099 102 102 102 170 170 170 F F F 5 100-119 104 104 97 173 173 162 T T T <--- 6 120-139 102 102 102 170 170 170 T T F <--- 7 140-159 104 104 103 173 173 172 T T F <--- 8 160-179 100 100 100 168 168 168 F F F 9 180-199 104 104 94 173 173 158 F F F <--- 10 200-219 104 104 101 173 173 169 F F F <--- 11 220-239 102 102 102 170 170 170 T T F <--- 12 240-259 102 102 102 170 170 170 F F F 13 260-279 104 104 102 173 173 170 T T T <--- 14 280-299 104 104 100 173 173 165 T T F <--- 15 300-319 98 98 98 163 163 163 T T F <--- 16 320-339 98 98 98 163 163 163 T T F <--- 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 102 102 170 170 170 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 103 103 103 172 172 172 F F F 22 440-459 103 103 103 172 172 172 F F F 23 460-479 103 103 103 172 172 172 F F F 24 480-499 103 103 103 172 172 172 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 100 100 100 168 168 168 F F F 27 540-559 102 102 102 171 171 171 F F F 28 560-579 102 102 102 171 171 171 T T F <--- 29 580-599 100 100 100 165 165 165 F F F Max gen: 104, Max flush: 173 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 105 105 105 174 174 174 F F F 2 040-059 101 101 101 169 169 169 F F F 3 060-079 102 102 102 170 170 170 F F F 4 080-099 102 102 102 170 170 170 F F F 5 100-119 105 105 105 174 174 174 F F F 6 120-139 105 105 105 174 174 174 F F F 7 140-159 105 105 105 174 174 174 F F F 8 160-179 100 100 100 168 168 168 F F F 9 180-199 104 104 104 173 173 173 F F F 10 200-219 104 104 104 173 173 173 F F F 11 220-239 105 105 105 174 174 174 F F F 12 240-259 102 102 102 170 170 170 F F F 13 260-279 105 105 105 174 174 174 F F F 14 280-299 105 105 105 174 174 174 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 102 102 170 170 170 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 103 103 103 172 172 172 F F F 22 440-459 103 103 103 172 172 172 F F F 23 460-479 103 103 103 172 172 172 F F F 24 480-499 103 103 103 172 172 172 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 100 100 100 168 168 168 F F F 27 540-559 102 102 102 171 171 171 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 105, Max flush: 174 Begin loop 53 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.972093021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.983831864Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984207625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984250654Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984267012Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984286061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"Crucible b0afdf6e-45e6-4f65-be26-27e1e44bb8bc has session id: 38187a1b-abc3-43c4-987a-dbc37d3ce430","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984641592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Upstairs opts: Upstairs UUID: b0afdf6e-45e6-4f65-be26-27e1e44bb8bc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984676638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Crucible stats registered with UUID: b0afdf6e-45e6-4f65-be26-27e1e44bb8bc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984692716Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:31.984705912Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.98485551Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984879441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.984893717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985011351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985025007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"b0afdf6e-45e6-4f65-be26-27e1e44bb8bc active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985037843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985050869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985064435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985080242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985093898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985107945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985121931Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"ds_connection connected from Ok(127.0.0.1:46049)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985166231Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"ds_connection connected from Ok(127.0.0.1:60457)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985181998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"ds_connection connected from Ok(127.0.0.1:40859)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985257683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985915792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985952049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.985968857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986294735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986311733Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[0]R flush_numbers[0..12]: [143, 174, 169, 170, 170, 174, 174, 174, 168, 173, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986346869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[0]R generation[0..12]: [84, 105, 101, 102, 102, 105, 105, 105, 100, 104, 104, 105]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986364116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986383855Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[1]R flush_numbers[0..12]: [143, 174, 169, 170, 170, 174, 174, 174, 168, 173, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986399773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[1]R generation[0..12]: [84, 105, 101, 102, 102, 105, 105, 105, 100, 104, 104, 105]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.98641549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986431827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[2]R flush_numbers[0..12]: [143, 174, 169, 170, 170, 174, 174, 174, 168, 173, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986447634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[2]R generation[0..12]: [84, 105, 101, 102, 102, 105, 105, 105, 100, 104, 104, 105]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986463131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986480129Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:04:31.986493925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986514495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Next flush: 175","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986528651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986541867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"":"downstairs","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986556004Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"0","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986582886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"1","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986596682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"client":"2","":"downstairs","session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986615831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986629767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"b0afdf6e-45e6-4f65-be26-27e1e44bb8bc is now active with session: 38187a1b-abc3-43c4-987a-dbc37d3ce430","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986644744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.98665721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629,"session_id":"38187a1b-abc3-43c4-987a-dbc37d3ce430"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986670656Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} {"msg":"Activated sub_volume b0afdf6e-45e6-4f65-be26-27e1e44bb8bc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:31.986682031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 83 len 20480 data: 6 7 7 9 7 02/30 Flush 03/30 Write block 62 len 28672 data: 7 7 8 8 9 9 10 04/30 Write block 124 len 24576 data: 11 12 14 16 14 13 05/30 Read block 328 len 12288 06/30 Read block 126 len 36864 07/30 Write block 216 len 20480 data: 8 8 7 7 10 08/30 Flush 09/30 Write block 216 len 20480 data: 9 9 8 8 11 10/30 Write block 151 len 28672 data: 10 10 9 9 9 10 9 11/30 Write block 285 len 28672 data: 8 10 11 11 14 12 12 12/30 Write block 103 len 24576 data: 5 5 3 5 6 10 13/30 Read block 471 len 20480 14/30 Write block 549 len 16384 data: 11 13 11 9 15/30 Read block 372 len 36864 16/30 Write block 429 len 4096 data: 11 17/30 Read block 420 len 12288 18/30 Write block 96 len 8192 data: 6 6 19/30 Read block 294 len 16384 20/30 Read block 151 len 40960 21/30 Flush 22/30 Write block 538 len 24576 data: 9 12 10 10 9 10 23/30 Read block 216 len 32768 24/30 Flush 25/30 Write block 50 len 32768 data: 11 13 11 13 12 11 9 7 26/30 Read block 110 len 8192 27/30 Read block 557 len 36864 28/30 Read block 517 len 12288 29/30 Read block 190 len 20480 30/30 Read block 216 len 8192 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 7 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 30 1 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:31.998893739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1629} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 105 105 105 174 174 174 F F F 2 040-059 101 101 101 169 169 169 T F T <--- 3 060-079 106 102 106 176 170 176 F F F <--- 4 080-099 106 102 106 177 170 177 F T F <--- 5 100-119 106 105 106 177 174 177 F F F <--- 6 120-139 106 105 106 176 174 176 F F F <--- 7 140-159 106 105 106 177 174 177 F F F <--- 8 160-179 100 100 100 168 168 168 F F F 9 180-199 104 104 104 173 173 173 F F F 10 200-219 106 104 106 177 173 177 F F F <--- 11 220-239 106 105 106 177 174 177 F F F <--- 12 240-259 102 102 102 170 170 170 F F F 13 260-279 105 105 105 174 174 174 F F F 14 280-299 106 105 106 177 174 177 F F F <--- 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 102 102 170 170 170 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 106 103 106 177 172 177 F F F <--- 22 440-459 103 103 103 172 172 172 F F F 23 460-479 103 103 103 172 172 172 F F F 24 480-499 103 103 103 172 172 172 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 106 100 106 178 168 178 F F F <--- 27 540-559 106 102 106 178 171 178 F F F <--- 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 106, Max flush: 178 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 F F F 1 020-039 105 105 105 174 174 174 F F F 2 040-059 107 107 107 179 179 179 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 106 106 106 177 177 177 F F F 5 100-119 106 106 106 177 177 177 F F F 6 120-139 106 106 106 176 176 176 F F F 7 140-159 106 106 106 177 177 177 F F F 8 160-179 100 100 100 168 168 168 F F F 9 180-199 104 104 104 173 173 173 F F F 10 200-219 106 106 106 177 177 177 F F F 11 220-239 106 106 106 177 177 177 F F F 12 240-259 102 102 102 170 170 170 F F F 13 260-279 105 105 105 174 174 174 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 102 102 102 170 170 170 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 106 106 106 177 177 177 F F F 22 440-459 103 103 103 172 172 172 F F F 23 460-479 103 103 103 172 172 172 F F F 24 480-499 103 103 103 172 172 172 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 106 106 106 178 178 178 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 107, Max flush: 179 Begin loop 54 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.322523946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334165974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334488221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334528029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334540385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334557933Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"Crucible 933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8 has session id: 6f971dec-94fa-41c7-9f3a-68dea51afcc5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334913305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Upstairs opts: Upstairs UUID: 933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.334983346Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Crucible stats registered with UUID: 933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335004356Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:36.335017632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335171092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335194893Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.3352098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335328784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335345311Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335375575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335389431Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335403518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335417574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.33543134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335445847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335460043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:64613)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335523832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:47063)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.33554024Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:35409)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.335566952Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336244921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336282388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336299686Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336655487Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336694205Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[0]R flush_numbers[0..12]: [143, 174, 179, 176, 177, 177, 176, 177, 168, 173, 177, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336762386Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[0]R generation[0..12]: [84, 105, 107, 106, 106, 106, 106, 106, 100, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336780504Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336797112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[1]R flush_numbers[0..12]: [143, 174, 179, 176, 177, 177, 176, 177, 168, 173, 177, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336813029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[1]R generation[0..12]: [84, 105, 107, 106, 106, 106, 106, 106, 100, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336828396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336844674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[2]R flush_numbers[0..12]: [143, 174, 179, 176, 177, 177, 176, 177, 168, 173, 177, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336860391Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[2]R generation[0..12]: [84, 105, 107, 106, 106, 106, 106, 106, 100, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336876218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336895467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336908953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:04:36.336923169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Next flush: 180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336940357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336953863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"":"downstairs","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.3369683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.336981076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"0","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337006947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"1","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337021364Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"client":"2","":"downstairs","session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337038762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337052048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8 is now active with session: 6f971dec-94fa-41c7-9f3a-68dea51afcc5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337067225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337079741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637,"session_id":"6f971dec-94fa-41c7-9f3a-68dea51afcc5"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337093067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} {"msg":"Activated sub_volume 933ea7cd-1b9c-4e22-b6bc-ecdbff79bbb8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:36.337104092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 53 len 8192 data: 14 13 02/30 Read block 249 len 16384 03/30 Read block 102 len 24576 04/30 Read block 368 len 12288 05/30 Read block 231 len 12288 06/30 Read block 321 len 12288 07/30 Read block 461 len 20480 08/30 Write block 475 len 32768 data: 9 7 9 10 11 10 11 10 09/30 Write block 138 len 12288 data: 11 10 9 10/30 Read block 338 len 16384 11/30 Flush 12/30 Write block 194 len 12288 data: 9 11 10 13/30 Read block 193 len 8192 14/30 Write block 84 len 32768 data: 8 8 10 8 7 9 12 13 15/30 Flush 16/30 Write block 100 len 36864 data: 5 6 6 6 6 4 6 7 11 17/30 Read block 20 len 32768 18/30 Read block 346 len 8192 19/30 Write block 197 len 16384 data: 8 5 5 5 20/30 Write block 138 len 28672 data: 12 11 10 7 7 8 9 21/30 Flush 22/30 Write block 381 len 12288 data: 11 14 16 23/30 Read block 315 len 32768 24/30 Read block 227 len 4096 25/30 Flush 26/30 Write block 110 len 8192 data: 10 10 27/30 Write block 423 len 40960 data: 10 10 12 11 11 12 12 12 12 13 28/30 Read block 557 len 36864 29/30 Write block 17 len 28672 data: 4 2 2 2 2 4 4 30/30 Read block 135 len 16384 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 1 26 Done 30 5 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:36.349640818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1637} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 84 84 84 143 143 143 T F T <--- 1 020-039 105 105 105 174 174 174 T F T <--- 2 040-059 108 107 108 180 179 180 F T F <--- 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 106 108 181 177 181 F F F <--- 5 100-119 108 106 108 182 177 182 T F T <--- 6 120-139 108 106 108 182 176 182 F F F <--- 7 140-159 108 106 108 182 177 182 F F F <--- 8 160-179 100 100 100 168 168 168 F F F 9 180-199 108 104 108 182 173 182 F F F <--- 10 200-219 108 106 108 182 177 182 F F F <--- 11 220-239 106 106 106 177 177 177 F F F 12 240-259 102 102 102 170 170 170 F F F 13 260-279 105 105 105 174 174 174 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 108 102 108 183 170 183 F F F <--- 20 400-419 102 102 102 170 170 170 F F F 21 420-439 106 106 106 177 177 177 T F T <--- 22 440-459 103 103 103 172 172 172 F F F 23 460-479 108 103 108 180 172 180 F F F <--- 24 480-499 108 103 108 180 172 180 F F F <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 106 106 106 178 178 178 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 108, Max flush: 183 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 F F F 1 020-039 109 109 109 184 184 184 F F F 2 040-059 108 108 108 180 180 180 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 108 108 181 181 181 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 108 108 108 182 182 182 F F F 7 140-159 108 108 108 182 182 182 F F F 8 160-179 100 100 100 168 168 168 F F F 9 180-199 108 108 108 182 182 182 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 106 106 106 177 177 177 F F F 12 240-259 102 102 102 170 170 170 F F F 13 260-279 105 105 105 174 174 174 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 F F F 19 380-399 108 108 108 183 183 183 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 109 109 109 184 184 184 F F F 22 440-459 103 103 103 172 172 172 F F F 23 460-479 108 108 108 180 180 180 F F F 24 480-499 108 108 108 180 180 180 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 106 106 106 178 178 178 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 109, Max flush: 184 Begin loop 55 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.67868867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.690387665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.690763096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.690805295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.690817941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.690836119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"Crucible 9c200d34-bc18-4841-9e66-090b5600a3cb has session id: 59263045-0aa9-44a4-bacf-1d8589d96ec4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691168601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Upstairs opts: Upstairs UUID: 9c200d34-bc18-4841-9e66-090b5600a3cb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691202566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Crucible stats registered with UUID: 9c200d34-bc18-4841-9e66-090b5600a3cb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691218653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:40.691232099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691393983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691418544Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691433521Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691555106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691572184Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"9c200d34-bc18-4841-9e66-090b5600a3cb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691589682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691605169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691619466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691633302Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.69167226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691714559Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.69173889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"ds_connection connected from Ok(127.0.0.1:43498)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691755007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"ds_connection connected from Ok(127.0.0.1:54199)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691770334Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"ds_connection connected from Ok(127.0.0.1:63655)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.691829131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692493123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.69252987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692547058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.69290384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692919758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[0]R flush_numbers[0..12]: [184, 184, 180, 176, 181, 184, 182, 182, 168, 182, 182, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692944449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[0]R generation[0..12]: [109, 109, 108, 106, 108, 109, 108, 108, 100, 108, 108, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692963057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692979685Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[1]R flush_numbers[0..12]: [184, 184, 180, 176, 181, 184, 182, 182, 168, 182, 182, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.692995482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[1]R generation[0..12]: [109, 109, 108, 106, 108, 109, 108, 108, 100, 108, 108, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693011369Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693027417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[2]R flush_numbers[0..12]: [184, 184, 180, 176, 181, 184, 182, 182, 168, 182, 182, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693042824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[2]R generation[0..12]: [109, 109, 108, 106, 108, 109, 108, 108, 100, 108, 108, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693058591Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693074548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693087844Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.6931017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Next flush: 185","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:04:40.693121139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693134795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"":"downstairs","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693149082Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693161578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"0","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693175524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"1","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693191361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"client":"2","":"downstairs","session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693205758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693218794Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"9c200d34-bc18-4841-9e66-090b5600a3cb is now active with session: 59263045-0aa9-44a4-bacf-1d8589d96ec4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693235891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693248377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645,"session_id":"59263045-0aa9-44a4-bacf-1d8589d96ec4"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693261893Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} {"msg":"Activated sub_volume 9c200d34-bc18-4841-9e66-090b5600a3cb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:40.693273168Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 185 len 20480 data: 10 9 9 8 12 02/30 Read block 269 len 20480 03/30 Read block 197 len 20480 04/30 Read block 214 len 12288 05/30 Write block 528 len 20480 data: 7 9 8 9 11 06/30 Write block 259 len 32768 data: 8 8 10 11 11 8 6 9 07/30 Read block 277 len 28672 08/30 Read block 371 len 36864 09/30 Read block 419 len 16384 10/30 Read block 170 len 24576 11/30 Write block 171 len 12288 data: 10 11 10 12/30 Read block 61 len 36864 13/30 Read block 583 len 12288 14/30 Read block 320 len 8192 15/30 Read block 587 len 36864 16/30 Write block 252 len 28672 data: 11 9 13 12 11 10 9 17/30 Write block 220 len 16384 data: 12 10 9 11 18/30 Read block 98 len 40960 19/30 Read block 406 len 16384 20/30 Read block 341 len 40960 21/30 Write block 436 len 16384 data: 11 11 13 13 22/30 Read block 476 len 36864 23/30 Write block 34 len 40960 data: 9 8 8 6 7 6 6 5 7 6 24/30 Write block 442 len 16384 data: 10 8 7 7 25/30 Read block 481 len 4096 26/30 Read block 244 len 8192 27/30 Write block 182 len 32768 data: 8 8 9 11 10 10 9 13 28/30 Write block 477 len 20480 data: 10 11 12 11 12 29/30 Read block 230 len 24576 30/30 Write block 378 len 24576 data: 10 12 13 12 15 17 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 9 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 29 29 7 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:40.705342301Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1645} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 F F F 1 020-039 109 109 109 184 184 184 T T F <--- 2 040-059 108 108 108 180 180 180 T T F <--- 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 108 108 181 181 181 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 108 108 108 182 182 182 F F F 7 140-159 108 108 108 182 182 182 F F F 8 160-179 100 100 100 168 168 168 T T F <--- 9 180-199 108 108 108 182 182 182 T T T 10 200-219 108 108 108 182 182 182 F F F 11 220-239 106 106 106 177 177 177 T T F <--- 12 240-259 102 102 102 170 170 170 T T T 13 260-279 105 105 105 174 174 174 T T T 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 96 96 96 160 160 160 T T F <--- 19 380-399 108 108 108 183 183 183 T T F <--- 20 400-419 102 102 102 170 170 170 F F F 21 420-439 109 109 109 184 184 184 T T F <--- 22 440-459 103 103 103 172 172 172 T T F <--- 23 460-479 108 108 108 180 180 180 T T F <--- 24 480-499 108 108 108 180 180 180 T T F <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 106 106 106 178 178 178 T T T 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 109, Max flush: 184 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 F F F 1 020-039 110 110 110 185 185 185 F F F 2 040-059 110 110 110 185 185 185 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 108 108 181 181 181 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 108 108 108 182 182 182 F F F 7 140-159 108 108 108 182 182 182 F F F 8 160-179 110 110 110 185 185 185 F F F 9 180-199 110 110 110 185 185 185 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 110 110 110 185 185 185 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 105 105 174 174 174 F F F 16 320-339 105 105 105 174 174 174 F F F 17 340-359 95 95 95 159 159 159 F F F 18 360-379 110 110 110 185 185 185 F F F 19 380-399 110 110 110 185 185 185 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 110 110 110 185 185 185 F F F 22 440-459 110 110 110 185 185 185 F F F 23 460-479 110 110 110 185 185 185 F F F 24 480-499 110 110 110 185 185 185 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 110 110 110 185 185 185 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 105 105 174 174 174 F F F 29 580-599 100 100 100 165 165 165 F F F Max gen: 110, Max flush: 185 Begin loop 56 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.042391776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:45.05303547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.053081491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.063520512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.063812735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.063851843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.063864009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.063881687Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"Crucible 79ffba71-4fbe-4967-abb7-c8756aaba5ec has session id: 80c02c6f-df5b-42a0-bb1a-72d24b03fedf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064224903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Upstairs opts: Upstairs UUID: 79ffba71-4fbe-4967-abb7-c8756aaba5ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064260889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Crucible stats registered with UUID: 79ffba71-4fbe-4967-abb7-c8756aaba5ec","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064277107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:45.064299947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064491904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064528261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064543888Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064651727Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064667234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"79ffba71-4fbe-4967-abb7-c8756aaba5ec active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064688464Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.06470136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064715666Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064729343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064743039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064757465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064771962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"ds_connection connected from Ok(127.0.0.1:52330)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064848867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"ds_connection connected from Ok(127.0.0.1:56710)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064874589Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"ds_connection connected from Ok(127.0.0.1:36011)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.064890466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.065589714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.065618147Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.065758651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.065947387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.065963144Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[0]R flush_numbers[0..12]: [184, 185, 185, 176, 181, 184, 182, 182, 185, 185, 182, 185]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066176741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[0]R generation[0..12]: [109, 110, 110, 106, 108, 109, 108, 108, 110, 110, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.06619542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066212207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[1]R flush_numbers[0..12]: [184, 185, 185, 176, 181, 184, 182, 182, 185, 185, 182, 185]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066227985Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[1]R generation[0..12]: [109, 110, 110, 106, 108, 109, 108, 108, 110, 110, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066243822Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066260389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[2]R flush_numbers[0..12]: [184, 185, 185, 176, 181, 184, 182, 182, 185, 185, 182, 185]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066279798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[2]R generation[0..12]: [109, 110, 110, 106, 108, 109, 108, 108, 110, 110, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066295615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066312243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Max found gen is 111","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066331102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Generation requested: 112 >= found:111","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066345498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Next flush: 186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066358634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.0663723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"":"downstairs","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066391469Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066404705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"0","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066419002Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1653,"client":"1","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.06643671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"client":"2","":"downstairs","session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066451076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066464362Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"79ffba71-4fbe-4967-abb7-c8756aaba5ec is now active with session: 80c02c6f-df5b-42a0-bb1a-72d24b03fedf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066479689Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066492595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653,"session_id":"80c02c6f-df5b-42a0-bb1a-72d24b03fedf"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066507522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} {"msg":"Activated sub_volume 79ffba71-4fbe-4967-abb7-c8756aaba5ec","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:45.066519057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 420 len 16384 data: 7 7 10 11 02/30 Write block 577 len 28672 data: 11 11 11 14 14 12 12 03/30 Write block 455 len 28672 data: 6 5 7 8 7 7 8 04/30 Read block 314 len 40960 05/30 Read block 365 len 8192 06/30 Read block 344 len 16384 07/30 Read block 126 len 28672 08/30 Read block 106 len 12288 09/30 Read block 310 len 4096 10/30 Write block 226 len 24576 data: 13 13 11 12 13 15 11/30 Read block 478 len 36864 12/30 Flush 13/30 Write block 420 len 8192 data: 8 8 14/30 Read block 342 len 8192 15/30 Write block 29 len 24576 data: 6 7 8 7 8 10 16/30 Flush 17/30 Write block 390 len 36864 data: 12 11 11 11 11 8 10 13 12 18/30 Write block 305 len 36864 data: 4 5 8 9 7 8 10 9 8 19/30 Read block 228 len 4096 20/30 Write block 23 len 16384 data: 5 7 7 7 21/30 Flush 22/30 Write block 333 len 4096 data: 7 23/30 Write block 169 len 20480 data: 8 10 11 12 11 24/30 Write block 163 len 40960 data: 6 6 7 8 8 8 9 11 12 13 25/30 Read block 486 len 8192 26/30 Read block 238 len 28672 27/30 Read block 342 len 40960 28/30 Write block 378 len 20480 data: 11 13 14 13 16 29/30 Write block 437 len 16384 data: 12 14 14 12 30/30 Write block 339 len 24576 data: 11 11 11 6 8 7 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Read 1 Done Done Done false 1009 Acked Write 6 Done Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 2 2 24 Done 10 28 28 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:45.078269295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1653} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 F F F 1 020-039 110 112 112 185 188 188 F F F <--- 2 040-059 110 110 110 185 185 185 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 108 108 181 181 181 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 108 108 108 182 182 182 F F F 7 140-159 108 108 108 182 182 182 F F F 8 160-179 110 110 110 185 185 185 F T T <--- 9 180-199 110 110 110 185 185 185 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 110 112 112 185 186 186 T F F <--- 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 105 112 112 174 188 188 F F F <--- 16 320-339 105 105 105 174 174 174 F T T <--- 17 340-359 95 95 95 159 159 159 F T T <--- 18 360-379 110 110 110 185 185 185 F T T <--- 19 380-399 110 112 112 185 188 188 F T T <--- 20 400-419 102 102 102 170 170 170 F F F 21 420-439 110 112 112 185 187 187 T T T <--- 22 440-459 110 112 112 185 186 186 T T T <--- 23 460-479 110 112 112 185 186 186 T F F <--- 24 480-499 110 110 110 185 185 185 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 110 110 110 185 185 185 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 105 112 112 174 186 186 T F F <--- 29 580-599 100 112 112 165 186 186 T F F <--- Max gen: 112, Max flush: 188 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 110 110 110 185 185 185 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 108 108 108 181 181 181 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 108 108 108 182 182 182 F F F 7 140-159 108 108 108 182 182 182 F F F 8 160-179 113 113 113 189 189 189 F F F 9 180-199 110 110 110 185 185 185 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 112 112 112 186 186 186 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 112 112 112 188 188 188 F F F 16 320-339 113 113 113 189 189 189 F F F 17 340-359 113 113 113 189 189 189 F F F 18 360-379 113 113 113 189 189 189 F F F 19 380-399 113 113 113 189 189 189 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 113 113 113 189 189 189 F F F 22 440-459 113 113 113 189 189 189 F F F 23 460-479 112 112 112 186 186 186 F F F 24 480-499 110 110 110 185 185 185 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 110 110 110 185 185 185 F F F 27 540-559 106 106 106 178 178 178 F F F 28 560-579 112 112 112 186 186 186 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 113, Max flush: 189 Begin loop 57 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.419539131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.431367734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.431694013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.431735452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.431748088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.431765846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"Crucible 407ec13b-3637-4dad-a321-4b5cdbbd4eaf has session id: 4ebfda42-0059-4a13-b31a-74640cce8213","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432157214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Upstairs opts: Upstairs UUID: 407ec13b-3637-4dad-a321-4b5cdbbd4eaf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432185227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Crucible stats registered with UUID: 407ec13b-3637-4dad-a321-4b5cdbbd4eaf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432199923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:49.432212829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432408868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.43243395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432451858Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432584548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432599545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"407ec13b-3637-4dad-a321-4b5cdbbd4eaf active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432620485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.43263304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432647167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432661133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432674579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432688906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432703202Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"ds_connection connected from Ok(127.0.0.1:54031)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432728474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"ds_connection connected from Ok(127.0.0.1:48281)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432743401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"ds_connection connected from Ok(127.0.0.1:36488)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.432822767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433477655Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433505948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433523436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433846673Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.43386267Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[0]R flush_numbers[0..12]: [184, 188, 185, 176, 181, 184, 182, 182, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433904269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[0]R generation[0..12]: [109, 112, 110, 106, 108, 109, 108, 108, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433921787Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433938415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[1]R flush_numbers[0..12]: [184, 188, 185, 176, 181, 184, 182, 182, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433954422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[1]R generation[0..12]: [109, 112, 110, 106, 108, 109, 108, 108, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433970449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.433987427Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[2]R flush_numbers[0..12]: [184, 188, 185, 176, 181, 184, 182, 182, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434003334Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[2]R generation[0..12]: [109, 112, 110, 106, 108, 109, 108, 108, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434018841Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434034868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434048515Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:04:49.434062671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Next flush: 190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.43408155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434097467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"":"downstairs","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434111883Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434124679Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"0","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434139196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"1","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434153552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"client":"2","":"downstairs","session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434168569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434184977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"407ec13b-3637-4dad-a321-4b5cdbbd4eaf is now active with session: 4ebfda42-0059-4a13-b31a-74640cce8213","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434200214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434212779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661,"session_id":"4ebfda42-0059-4a13-b31a-74640cce8213"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434226125Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} {"msg":"Activated sub_volume 407ec13b-3637-4dad-a321-4b5cdbbd4eaf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:49.434237511Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Write block 326 len 24576 data: 12 10 10 10 9 7 04/30 Read block 229 len 28672 05/30 Read block 217 len 12288 06/30 Flush 07/30 Write block 553 len 32768 data: 13 11 10 9 7 7 8 6 08/30 Read block 263 len 24576 09/30 Read block 556 len 12288 10/30 Read block 124 len 16384 11/30 Read block 120 len 16384 12/30 Read block 40 len 28672 13/30 Flush 14/30 Write block 523 len 40960 data: 6 6 7 6 5 8 10 9 10 12 15/30 Read block 241 len 24576 16/30 Read block 174 len 32768 17/30 Write block 84 len 40960 data: 9 9 11 9 8 10 13 14 12 10 18/30 Read block 80 len 40960 19/30 Write block 128 len 32768 data: 15 14 13 17 14 15 14 14 20/30 Flush 21/30 Write block 539 len 24576 data: 13 11 11 10 11 11 22/30 Read block 333 len 40960 23/30 Write block 13 len 28672 data: 8 5 5 5 5 3 3 24/30 Write block 44 len 24576 data: 8 8 10 11 13 11 25/30 Read block 218 len 36864 26/30 Write block 445 len 16384 data: 8 6 4 8 27/30 Read block 31 len 24576 28/30 Read block 395 len 28672 29/30 Write block 488 len 24576 data: 6 10 10 9 8 10 30/30 Write block 147 len 24576 data: 7 8 10 11 11 11 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 6 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 26 2 30 Done 26 2 26 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1001)) Some(JobId(1019)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:49.447794611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1661} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 184 184 184 T F T <--- 1 020-039 112 112 112 188 188 188 F F F 2 040-059 110 110 110 185 185 185 T F T <--- 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 108 114 194 181 194 F F F <--- 5 100-119 109 109 109 184 184 184 F F F 6 120-139 114 108 114 194 182 194 F F F <--- 7 140-159 108 108 108 182 182 182 T F T <--- 8 160-179 113 113 113 189 189 189 F F F 9 180-199 110 110 110 185 185 185 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 112 112 112 186 186 186 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 112 112 112 188 188 188 F F F 16 320-339 114 113 114 192 189 192 F T F <--- 17 340-359 113 113 113 189 189 189 F F F 18 360-379 113 113 113 189 189 189 F F F 19 380-399 113 113 113 189 189 189 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 113 113 113 189 189 189 F F F 22 440-459 113 113 113 189 189 189 T F T <--- 23 460-479 112 112 112 186 186 186 F F F 24 480-499 110 110 110 185 185 185 T F T <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 114 110 114 194 185 194 T F T <--- 27 540-559 114 106 114 193 178 193 T F T <--- 28 560-579 114 112 114 193 186 193 F F F <--- 29 580-599 112 112 112 186 186 186 F F F Max gen: 114, Max flush: 194 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 115 115 115 195 195 195 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 114 114 194 194 194 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 114 114 114 194 194 194 F F F 7 140-159 115 115 115 195 195 195 F F F 8 160-179 113 113 113 189 189 189 F F F 9 180-199 110 110 110 185 185 185 F F F 10 200-219 108 108 108 182 182 182 F F F 11 220-239 112 112 112 186 186 186 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 112 112 112 188 188 188 F F F 16 320-339 114 114 114 192 192 192 F F F 17 340-359 113 113 113 189 189 189 F F F 18 360-379 113 113 113 189 189 189 F F F 19 380-399 113 113 113 189 189 189 F F F 20 400-419 102 102 102 170 170 170 F F F 21 420-439 113 113 113 189 189 189 F F F 22 440-459 115 115 115 195 195 195 F F F 23 460-479 112 112 112 186 186 186 F F F 24 480-499 115 115 115 195 195 195 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 115 115 115 195 195 195 F F F 28 560-579 114 114 114 193 193 193 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 115, Max flush: 195 Begin loop 58 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.775572926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:53.786224323Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.786279499Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.796736918Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797087948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797127105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797143143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797161111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"Crucible 9997dc74-2da6-4d49-9c9a-1b601fe0bd41 has session id: 41d3d2ce-d294-45aa-b03c-cf25b7169d0f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797507118Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Upstairs opts: Upstairs UUID: 9997dc74-2da6-4d49-9c9a-1b601fe0bd41, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797541404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Crucible stats registered with UUID: 9997dc74-2da6-4d49-9c9a-1b601fe0bd41","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797557261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:53.797570777Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.7977308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797756722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797771309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797900467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797917385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"9997dc74-2da6-4d49-9c9a-1b601fe0bd41 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797935383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797948729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797962936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.797976882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.7979934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798008276Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798022713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54982)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.79806101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43784)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798078068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:61879)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798092885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798870119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798898342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.798920081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.79924575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799270811Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[0]R flush_numbers[0..12]: [195, 188, 195, 176, 194, 184, 194, 195, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799342634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[0]R generation[0..12]: [115, 112, 115, 106, 114, 109, 114, 115, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799360812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.79937746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[1]R flush_numbers[0..12]: [195, 188, 195, 176, 194, 184, 194, 195, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799396768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[1]R generation[0..12]: [115, 112, 115, 106, 114, 109, 114, 115, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799412606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799429123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[2]R flush_numbers[0..12]: [195, 188, 195, 176, 194, 184, 194, 195, 189, 185, 182, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.79944488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[2]R generation[0..12]: [115, 112, 115, 106, 114, 109, 114, 115, 113, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799460257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799476615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799490381Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.79951038Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Next flush: 196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799523666Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799537452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"":"downstairs","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799551779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799565005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"0","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799579372Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"1","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "crucible","level":30,"time":"2026-03-14T05:04:53.799596339Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"client":"2","":"downstairs","session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799610596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799625733Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"9997dc74-2da6-4d49-9c9a-1b601fe0bd41 is now active with session: 41d3d2ce-d294-45aa-b03c-cf25b7169d0f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799640669Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799653405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669,"session_id":"41d3d2ce-d294-45aa-b03c-cf25b7169d0f"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799666741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} {"msg":"Activated sub_volume 9997dc74-2da6-4d49-9c9a-1b601fe0bd41","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:53.799678006Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Write block 190 len 40960 data: 10 12 9 7 10 12 11 9 6 6 04/30 Write block 462 len 4096 data: 7 05/30 Read block 241 len 12288 06/30 Read block 473 len 32768 07/30 Write block 561 len 28672 data: 6 5 7 9 9 10 8 08/30 Read block 515 len 28672 09/30 Write block 212 len 20480 data: 9 8 8 8 10 10/30 Read block 313 len 16384 11/30 Read block 410 len 8192 12/30 Write block 199 len 24576 data: 7 6 3 3 3 3 13/30 Write block 414 len 8192 data: 7 8 14/30 Read block 580 len 12288 15/30 Write block 566 len 20480 data: 11 9 10 13 13 16/30 Read block 274 len 36864 17/30 Read block 79 len 4096 18/30 Read block 558 len 36864 19/30 Read block 234 len 32768 20/30 Write block 360 len 4096 data: 8 21/30 Read block 354 len 12288 22/30 Write block 424 len 12288 data: 11 13 12 23/30 Write block 159 len 4096 data: 7 24/30 Read block 370 len 12288 25/30 Write block 199 len 20480 data: 8 7 4 4 4 26/30 Read block 227 len 28672 27/30 Read block 226 len 20480 28/30 Write block 0 len 24576 data: 3 3 4 6 6 9 29/30 Write block 400 len 16384 data: 10 9 7 4 30/30 Write block 130 len 32768 data: 14 18 15 16 15 15 13 10 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 10 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 3 26 26 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:53.811118774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1669} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 115 115 115 195 195 195 F T T <--- 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 114 114 194 194 194 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 114 114 114 194 194 194 F T T <--- 7 140-159 115 115 115 195 195 195 F T T <--- 8 160-179 113 113 113 189 189 189 F F F 9 180-199 110 110 110 185 185 185 T T T 10 200-219 108 108 108 182 182 182 F T T <--- 11 220-239 112 112 112 186 186 186 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 112 112 112 188 188 188 F F F 16 320-339 114 114 114 192 192 192 F F F 17 340-359 113 113 113 189 189 189 F F F 18 360-379 113 113 113 189 189 189 F T T <--- 19 380-399 113 113 113 189 189 189 F F F 20 400-419 102 102 102 170 170 170 F T T <--- 21 420-439 113 113 113 189 189 189 F T T <--- 22 440-459 115 115 115 195 195 195 F F F 23 460-479 112 112 112 186 186 186 T T T 24 480-499 115 115 115 195 195 195 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 115 115 115 195 195 195 F F F 28 560-579 114 114 114 193 193 193 F T T <--- 29 580-599 112 112 112 186 186 186 F F F Max gen: 115, Max flush: 195 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 202c202 < 6, --- > 8, diff found after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 114 114 194 194 194 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 116 116 116 196 196 196 F F F 7 140-159 116 116 116 196 196 196 F F F 8 160-179 113 113 113 189 189 189 F F F 9 180-199 116 116 116 196 196 196 F F F 10 200-219 116 116 116 196 196 196 F F F 11 220-239 112 112 112 186 186 186 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F F F 14 280-299 106 106 106 177 177 177 F F F 15 300-319 112 112 112 188 188 188 F F F 16 320-339 114 114 114 192 192 192 F F F 17 340-359 113 113 113 189 189 189 F F F 18 360-379 116 116 116 196 196 196 F F F 19 380-399 113 113 113 189 189 189 F F F 20 400-419 116 116 116 196 196 196 F F F 21 420-439 116 116 116 196 196 196 F F F 22 440-459 115 115 115 195 195 195 F F F 23 460-479 116 116 116 196 196 196 F F F 24 480-499 115 115 115 195 195 195 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 115 115 115 195 195 195 F F F 28 560-579 116 116 116 196 196 196 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 116, Max flush: 196 Begin loop 59 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.153914343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:58.164661514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.16471845Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175038998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175375941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175415949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175427665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175445583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"Crucible 51f51469-0663-4082-b5af-ee7c6d60353a has session id: 05fe3c01-7775-41dc-b786-2f2aa5952b91","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175772522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Upstairs opts: Upstairs UUID: 51f51469-0663-4082-b5af-ee7c6d60353a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175798493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Crucible stats registered with UUID: 51f51469-0663-4082-b5af-ee7c6d60353a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.175815131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:58.175828347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176000305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176023266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176037672Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176159348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176176415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"51f51469-0663-4082-b5af-ee7c6d60353a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176193603Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176206089Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176220235Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176234132Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176247498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176267877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176282183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"ds_connection connected from Ok(127.0.0.1:38045)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176306234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"ds_connection connected from Ok(127.0.0.1:38632)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176321421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"ds_connection connected from Ok(127.0.0.1:55934)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.176470229Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177139984Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177167556Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177183974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177521097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177562546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[0]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 196, 189, 196, 196, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177605856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[0]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 116, 113, 116, 116, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177622464Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177639241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[1]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 196, 189, 196, 196, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177654988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[1]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 116, 113, 116, 116, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177670625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177687063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[2]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 196, 189, 196, 196, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.17770286Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[2]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 116, 113, 116, 116, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177718717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177738216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 117","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177758385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Generation requested: 118 >= found:117","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177772532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Next flush: 197","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177785788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177799174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"":"downstairs","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.17781304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177825596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"0","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177840223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"client":"1","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177854749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 client":"2","":"downstairs","session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177872547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177886154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"51f51469-0663-4082-b5af-ee7c6d60353a is now active with session: 05fe3c01-7775-41dc-b786-2f2aa5952b91","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.17790146Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177913956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680,"session_id":"05fe3c01-7775-41dc-b786-2f2aa5952b91"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177927452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} {"msg":"Activated sub_volume 51f51469-0663-4082-b5af-ee7c6d60353a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:58.177939008Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 305 len 20480 data: 5 6 9 10 8 02/30 Flush 03/30 Flush 04/30 Write block 337 len 40960 data: 11 12 12 12 12 7 9 8 8 8 05/30 Read block 321 len 12288 06/30 Write block 325 len 8192 data: 10 13 07/30 Read block 224 len 24576 08/30 Flush 09/30 Write block 231 len 36864 data: 16 14 13 12 11 12 10 8 5 10/30 Read block 67 len 28672 11/30 Write block 156 len 36864 data: 11 10 9 8 4 3 4 7 7 12/30 Flush 13/30 Write block 197 len 12288 data: 10 7 7 14/30 Write block 484 len 36864 data: 6 7 6 6 7 11 11 10 9 15/30 Flush 16/30 Write block 278 len 28672 data: 11 10 8 8 10 8 7 17/30 Write block 284 len 24576 data: 8 9 11 12 12 15 18/30 Write block 544 len 4096 data: 12 19/30 Read block 83 len 12288 20/30 Read block 457 len 32768 21/30 Read block 354 len 36864 22/30 Write block 276 len 8192 data: 12 12 23/30 Write block 540 len 12288 data: 12 12 11 24/30 Read block 430 len 20480 25/30 Read block 57 len 20480 26/30 Read block 178 len 32768 27/30 Write block 399 len 4096 data: 12 28/30 Write block 435 len 24576 data: 8 12 13 15 15 13 29/30 Write block 228 len 32768 data: 12 13 14 17 15 14 13 12 30/30 Write block 354 len 16384 data: 10 10 9 8 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 9 Done Done Done false 1009 Acked Read 7 Done Done Done false 1010 Acked Write 9 Done Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 19 2 2 23 Done 3 20 20 43 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:58.190089628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1680} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 114 114 194 194 194 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 116 116 116 196 196 196 F F F 7 140-159 116 118 118 196 200 200 T F F <--- 8 160-179 113 118 118 189 200 200 T F F <--- 9 180-199 116 118 118 196 201 201 F F F <--- 10 200-219 116 116 116 196 196 196 F F F 11 220-239 112 118 118 186 200 200 T T T <--- 12 240-259 110 110 110 185 185 185 F F F 13 260-279 110 110 110 185 185 185 F T T <--- 14 280-299 106 106 106 177 177 177 F T T <--- 15 300-319 118 118 118 197 197 197 F F F 16 320-339 118 118 118 199 199 199 F F F 17 340-359 118 118 118 199 199 199 F T T <--- 18 360-379 116 116 116 196 196 196 F F F 19 380-399 113 113 113 189 189 189 F T T <--- 20 400-419 116 116 116 196 196 196 F F F 21 420-439 116 116 116 196 196 196 F T T <--- 22 440-459 115 115 115 195 195 195 F T T <--- 23 460-479 116 116 116 196 196 196 F F F 24 480-499 115 118 118 195 201 201 F F F <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 115 115 115 195 195 195 F T T <--- 28 560-579 116 116 116 196 196 196 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 118, Max flush: 201 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 114 114 114 194 194 194 F F F 5 100-119 109 109 109 184 184 184 F F F 6 120-139 116 116 116 196 196 196 F F F 7 140-159 118 118 118 200 200 200 F F F 8 160-179 118 118 118 200 200 200 F F F 9 180-199 118 118 118 201 201 201 F F F 10 200-219 116 116 116 196 196 196 F F F 11 220-239 119 119 119 202 202 202 F F F 12 240-259 110 110 110 185 185 185 F F F 13 260-279 119 119 119 202 202 202 F F F 14 280-299 119 119 119 202 202 202 F F F 15 300-319 118 118 118 197 197 197 F F F 16 320-339 118 118 118 199 199 199 F F F 17 340-359 119 119 119 202 202 202 F F F 18 360-379 116 116 116 196 196 196 F F F 19 380-399 119 119 119 202 202 202 F F F 20 400-419 116 116 116 196 196 196 F F F 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 116 116 116 196 196 196 F F F 24 480-499 118 118 118 201 201 201 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 119 119 119 202 202 202 F F F 28 560-579 116 116 116 196 196 196 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 119, Max flush: 202 Begin loop 60 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.529934013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.541870956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542203488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542249088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542261904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542280453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"Crucible 93ba5f4b-36bd-4bf3-a950-a8dd50aed774 has session id: c30b814a-e7ab-4164-bf14-c4b2440888ba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542607301Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Upstairs opts: Upstairs UUID: 93ba5f4b-36bd-4bf3-a950-a8dd50aed774, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542641917Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Crucible stats registered with UUID: 93ba5f4b-36bd-4bf3-a950-a8dd50aed774","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542657834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:02.54267126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542828992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542852023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.542866769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.54299938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543032515Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"93ba5f4b-36bd-4bf3-a950-a8dd50aed774 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543045761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543058577Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543075374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543089701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543103447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543117654Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.54313185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:38740)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543156241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:45984)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543171168Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:41304)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543304299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543862593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543952804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.543985269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544243246Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544282724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[0]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 200, 200, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544394085Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[0]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 118, 118, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544411262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544431442Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[1]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 200, 200, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544448029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[1]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 118, 118, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544463776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544480034Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[2]R flush_numbers[0..12]: [196, 188, 195, 176, 194, 184, 196, 200, 200, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544495961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[2]R generation[0..12]: [116, 112, 115, 106, 114, 109, 116, 118, 118, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544511338Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544527825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544546644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544560911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Next flush: 203","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544574177Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544587403Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"":"downstairs","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544601589Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544614465Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"0","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544628401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"1","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:05:02.544644359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"client":"2","":"downstairs","session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544661586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544675423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"93ba5f4b-36bd-4bf3-a950-a8dd50aed774 is now active with session: c30b814a-e7ab-4164-bf14-c4b2440888ba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544689919Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544702525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688,"session_id":"c30b814a-e7ab-4164-bf14-c4b2440888ba"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544715701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} {"msg":"Activated sub_volume 93ba5f4b-36bd-4bf3-a950-a8dd50aed774","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:02.544726976Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 241 len 36864 data: 4 4 6 8 9 12 13 13 14 02/30 Write block 383 len 4096 data: 18 03/30 Write block 84 len 40960 data: 10 10 12 10 9 11 14 15 13 11 04/30 Read block 426 len 24576 05/30 Read block 206 len 32768 06/30 Write block 282 len 36864 data: 11 9 9 10 12 13 13 16 13 07/30 Write block 118 len 16384 data: 6 7 7 8 08/30 Read block 478 len 4096 09/30 Read block 186 len 4096 10/30 Read block 456 len 24576 11/30 Read block 201 len 24576 12/30 Read block 41 len 4096 13/30 Read block 581 len 28672 14/30 Read block 137 len 28672 15/30 Write block 161 len 4096 data: 4 16/30 Read block 223 len 36864 17/30 Read block 288 len 40960 18/30 Write block 148 len 4096 data: 9 19/30 Write block 89 len 36864 data: 12 15 16 14 12 9 7 7 7 20/30 Write block 100 len 4096 data: 6 21/30 Read block 378 len 8192 22/30 Read block 254 len 4096 23/30 Read block 375 len 32768 24/30 Flush 25/30 Flush 26/30 Write block 480 len 4096 data: 12 27/30 Read block 117 len 32768 28/30 Write block 413 len 12288 data: 11 8 9 29/30 Read block 351 len 12288 30/30 Read block 573 len 16384 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 1 Done Done Sent false 1002 Acked Write 10 Done Done Sent false 1003 Acked Read 6 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:02.555861734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1688} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 120 120 114 203 203 194 F F F <--- 5 100-119 120 120 109 203 203 184 F F F <--- 6 120-139 120 120 116 203 203 196 F F F <--- 7 140-159 120 120 118 203 203 200 F F F <--- 8 160-179 120 120 118 203 203 200 F F F <--- 9 180-199 118 118 118 201 201 201 F F F 10 200-219 116 116 116 196 196 196 F F F 11 220-239 119 119 119 202 202 202 F F F 12 240-259 120 120 110 203 203 185 F F T <--- 13 260-279 119 119 119 202 202 202 F F F 14 280-299 120 120 119 203 203 202 F F F <--- 15 300-319 118 118 118 197 197 197 F F F 16 320-339 118 118 118 199 199 199 F F F 17 340-359 119 119 119 202 202 202 F F F 18 360-379 116 116 116 196 196 196 F F F 19 380-399 120 120 119 203 203 202 F F F <--- 20 400-419 116 116 116 196 196 196 T T F <--- 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 116 116 116 196 196 196 F F F 24 480-499 118 118 118 201 201 201 T T F <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 119 119 119 202 202 202 F F F 28 560-579 116 116 116 196 196 196 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 120, Max flush: 203 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 106 106 106 176 176 176 F F F 4 080-099 120 120 120 203 203 203 F F F 5 100-119 120 120 120 203 203 203 F F F 6 120-139 120 120 120 203 203 203 F F F 7 140-159 120 120 120 203 203 203 F F F 8 160-179 120 120 120 203 203 203 F F F 9 180-199 118 118 118 201 201 201 F F F 10 200-219 116 116 116 196 196 196 F F F 11 220-239 119 119 119 202 202 202 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 119 119 119 202 202 202 F F F 14 280-299 120 120 120 203 203 203 F F F 15 300-319 118 118 118 197 197 197 F F F 16 320-339 118 118 118 199 199 199 F F F 17 340-359 119 119 119 202 202 202 F F F 18 360-379 116 116 116 196 196 196 F F F 19 380-399 120 120 120 203 203 203 F F F 20 400-419 121 121 121 204 204 204 F F F 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 116 116 116 196 196 196 F F F 24 480-499 121 121 121 204 204 204 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 F F F 27 540-559 119 119 119 202 202 202 F F F 28 560-579 116 116 116 196 196 196 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 121, Max flush: 204 Begin loop 61 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.876189107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.887815318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888165868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888206566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888218482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.88823612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"Crucible 41b559de-08f6-41c2-9f5c-4a4fca0bd04a has session id: c2df84a2-c05c-4bc0-9251-f29d9a804385","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888612361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Upstairs opts: Upstairs UUID: 41b559de-08f6-41c2-9f5c-4a4fca0bd04a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888646797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Crucible stats registered with UUID: 41b559de-08f6-41c2-9f5c-4a4fca0bd04a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888665805Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:06.888679261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.88885126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888885335Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.888901302Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889014284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889031152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"41b559de-08f6-41c2-9f5c-4a4fca0bd04a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.88904877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889061585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889075442Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889089088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.88913851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889162551Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889177558Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"ds_connection connected from Ok(127.0.0.1:42935)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889192635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"ds_connection connected from Ok(127.0.0.1:58393)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.88931308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"ds_connection connected from Ok(127.0.0.1:61325)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889344654Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889931481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889967848Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.889985256Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890334245Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890352203Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[0]R flush_numbers[0..12]: [196, 188, 195, 176, 203, 203, 203, 203, 203, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890376594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[0]R generation[0..12]: [116, 112, 115, 106, 120, 120, 120, 120, 120, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890392501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890409149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[1]R flush_numbers[0..12]: [196, 188, 195, 176, 203, 203, 203, 203, 203, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890425576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[1]R generation[0..12]: [116, 112, 115, 106, 120, 120, 120, 120, 120, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890441293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890457711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[2]R flush_numbers[0..12]: [196, 188, 195, 176, 203, 203, 203, 203, 203, 201, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890473788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[2]R generation[0..12]: [116, 112, 115, 106, 120, 120, 120, 120, 120, 118, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890489605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890509054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890523001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:06.890536987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Next flush: 205","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890554995Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890568531Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"":"downstairs","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890582508Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890595063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"0","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.89060926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"1","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890623286Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"client":"2","":"downstairs","session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890637663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 time":"2026-03-14T05:05:06.890651009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"41b559de-08f6-41c2-9f5c-4a4fca0bd04a is now active with session: c2df84a2-c05c-4bc0-9251-f29d9a804385","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890668887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890681713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696,"session_id":"c2df84a2-c05c-4bc0-9251-f29d9a804385"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890694979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} {"msg":"Activated sub_volume 41b559de-08f6-41c2-9f5c-4a4fca0bd04a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:06.890706744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 335 len 12288 data: 8 9 12 02/30 Write block 114 len 40960 data: 6 7 7 7 7 8 8 9 9 9 03/30 Read block 532 len 40960 04/30 Write block 570 len 12288 data: 14 11 11 05/30 Write block 74 len 28672 data: 9 12 11 12 11 11 12 06/30 Write block 279 len 12288 data: 11 9 9 07/30 Read block 521 len 36864 08/30 Read block 81 len 20480 09/30 Write block 479 len 4096 data: 13 10/30 Read block 235 len 20480 11/30 Read block 97 len 40960 12/30 Write block 488 len 28672 data: 8 12 12 11 10 11 13 13/30 Read block 365 len 40960 14/30 Write block 402 len 40960 data: 8 5 4 4 6 10 10 11 10 11 15/30 Read block 357 len 24576 16/30 Read block 434 len 12288 17/30 Write block 574 len 8192 data: 12 13 18/30 Write block 166 len 40960 data: 9 9 9 10 12 13 14 12 10 8 19/30 Write block 214 len 16384 data: 9 9 11 10 20/30 Flush 21/30 Write block 480 len 40960 data: 13 13 11 9 7 8 7 7 9 13 22/30 Flush 23/30 Write block 139 len 32768 data: 12 11 8 8 9 10 8 8 24/30 Flush 25/30 Write block 188 len 20480 data: 10 14 11 13 10 26/30 Read block 287 len 16384 27/30 Write block 527 len 16384 data: 6 9 11 10 28/30 Read block 362 len 16384 29/30 Read block 389 len 24576 30/30 Write block 357 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Sent false 1001 Acked Write 10 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:06.903436768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1696} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 122 122 106 205 205 176 F F F <--- 4 080-099 122 122 120 205 205 203 F F F <--- 5 100-119 122 122 120 205 205 203 F F F <--- 6 120-139 122 122 120 207 207 203 F F F <--- 7 140-159 122 122 120 207 207 203 F F F <--- 8 160-179 122 122 120 205 205 203 F F F <--- 9 180-199 118 118 118 201 201 201 T T F <--- 10 200-219 122 122 116 205 205 196 F F F <--- 11 220-239 119 119 119 202 202 202 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 122 122 119 205 205 202 F F F <--- 14 280-299 122 122 120 205 205 203 F F F <--- 15 300-319 118 118 118 197 197 197 F F F 16 320-339 122 122 118 205 205 199 F F F <--- 17 340-359 119 119 119 202 202 202 T T F <--- 18 360-379 116 116 116 196 196 196 F F F 19 380-399 120 120 120 203 203 203 F F F 20 400-419 122 122 121 205 205 204 F F F <--- 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 122 122 116 205 205 196 F F F <--- 24 480-499 122 122 121 206 206 204 F F F <--- 25 500-519 100 100 100 168 168 168 F F F 26 520-539 115 115 115 195 195 195 T T F <--- 27 540-559 119 119 119 202 202 202 F F F 28 560-579 122 122 116 205 205 196 F F F <--- 29 580-599 112 112 112 186 186 186 F F F Max gen: 122, Max flush: 207 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 122 122 122 205 205 205 F F F 4 080-099 122 122 122 205 205 205 F F F 5 100-119 122 122 122 205 205 205 F F F 6 120-139 122 122 122 207 207 207 F F F 7 140-159 122 122 122 207 207 207 F F F 8 160-179 122 122 122 205 205 205 F F F 9 180-199 123 123 123 208 208 208 F F F 10 200-219 122 122 122 205 205 205 F F F 11 220-239 119 119 119 202 202 202 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 122 122 122 205 205 205 F F F 14 280-299 122 122 122 205 205 205 F F F 15 300-319 118 118 118 197 197 197 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 123 123 123 208 208 208 F F F 18 360-379 116 116 116 196 196 196 F F F 19 380-399 120 120 120 203 203 203 F F F 20 400-419 122 122 122 205 205 205 F F F 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 122 122 122 205 205 205 F F F 24 480-499 122 122 122 206 206 206 F F F 25 500-519 100 100 100 168 168 168 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 119 119 119 202 202 202 F F F 28 560-579 122 122 122 205 205 205 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 123, Max flush: 208 Begin loop 62 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.240612681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252227627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252544711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252583749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252595915Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252613963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"Crucible 3af1e4c4-54be-4382-9e40-79b65bbcf5b7 has session id: 14d73b00-671e-4847-be7d-9bad3da6d462","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.252977038Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Upstairs opts: Upstairs UUID: 3af1e4c4-54be-4382-9e40-79b65bbcf5b7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253004191Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Crucible stats registered with UUID: 3af1e4c4-54be-4382-9e40-79b65bbcf5b7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253019207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:11.253032453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253220089Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253255165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253270932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253378171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253393148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"3af1e4c4-54be-4382-9e40-79b65bbcf5b7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253412116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253425312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253439389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253453385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253466771Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.25348634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253500977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"ds_connection connected from Ok(127.0.0.1:32859)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253534902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"ds_connection connected from Ok(127.0.0.1:57200)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253550879Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"ds_connection connected from Ok(127.0.0.1:42790)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.253565706Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254317288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254347652Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.25436484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254699602Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254746353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[0]R flush_numbers[0..12]: [196, 188, 195, 205, 205, 205, 207, 207, 205, 208, 205, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254790944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[0]R generation[0..12]: [116, 112, 115, 122, 122, 122, 122, 122, 122, 123, 122, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254808622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254825299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[1]R flush_numbers[0..12]: [196, 188, 195, 205, 205, 205, 207, 207, 205, 208, 205, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254841257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[1]R generation[0..12]: [116, 112, 115, 122, 122, 122, 122, 122, 122, 123, 122, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254856884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254872931Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[2]R flush_numbers[0..12]: [196, 188, 195, 205, 205, 205, 207, 207, 205, 208, 205, 202]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254888598Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[2]R generation[0..12]: [116, 112, 115, 122, 122, 122, 122, 122, 122, 123, 122, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254904545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254920593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:11.254934059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254954488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Next flush: 209","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254967644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.25498096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"":"downstairs","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.254995027Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255009383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"0","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.25502374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"1","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255037626Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"client":"2","":"downstairs","session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255054174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.25506795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"3af1e4c4-54be-4382-9e40-79b65bbcf5b7 is now active with session: 14d73b00-671e-4847-be7d-9bad3da6d462","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255082947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255095422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704,"session_id":"14d73b00-671e-4847-be7d-9bad3da6d462"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255109049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} {"msg":"Activated sub_volume 3af1e4c4-54be-4382-9e40-79b65bbcf5b7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:11.255120844Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 359 len 24576 data: 9 9 6 7 7 10 02/30 Write block 197 len 24576 data: 11 8 8 8 5 5 03/30 Read block 197 len 8192 04/30 Write block 560 len 12288 data: 7 7 6 05/30 Read block 128 len 16384 06/30 Read block 20 len 28672 07/30 Write block 276 len 28672 data: 13 13 12 12 10 10 12 08/30 Write block 215 len 28672 data: 10 12 11 9 9 13 11 09/30 Read block 467 len 8192 10/30 Read block 221 len 40960 11/30 Flush 12/30 Write block 151 len 40960 data: 12 12 10 10 10 12 11 10 9 5 13/30 Read block 353 len 4096 14/30 Write block 550 len 12288 data: 14 12 10 15/30 Read block 62 len 8192 16/30 Write block 107 len 20480 data: 8 12 9 11 11 17/30 Read block 304 len 12288 18/30 Flush 19/30 Write block 501 len 4096 data: 10 20/30 Write block 347 len 16384 data: 7 9 11 9 21/30 Flush 22/30 Write block 187 len 32768 data: 11 11 15 12 14 11 8 11 23/30 Write block 95 len 28672 data: 8 8 8 7 6 7 7 24/30 Read block 156 len 36864 25/30 Read block 259 len 20480 26/30 Write block 356 len 16384 data: 10 10 11 10 27/30 Write block 127 len 12288 data: 17 16 15 28/30 Write block 34 len 4096 data: 11 29/30 Read block 218 len 12288 30/30 Write block 167 len 32768 data: 10 10 11 13 14 15 13 11 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:11.2669773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1704} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 112 112 112 188 188 188 T F T <--- 2 040-059 115 115 115 195 195 195 F F F 3 060-079 122 122 122 205 205 205 F F F 4 080-099 122 122 122 205 205 205 T F T <--- 5 100-119 124 122 124 210 205 210 T F T <--- 6 120-139 122 122 122 207 207 207 T F T <--- 7 140-159 124 122 124 210 207 210 F F F <--- 8 160-179 124 122 124 210 205 210 T F T <--- 9 180-199 124 123 124 209 208 209 T T T <--- 10 200-219 124 122 124 209 205 209 F T F <--- 11 220-239 124 119 124 209 202 209 F F F <--- 12 240-259 120 120 120 203 203 203 F F F 13 260-279 124 122 124 209 205 209 F F F <--- 14 280-299 124 122 124 209 205 209 F F F <--- 15 300-319 118 118 118 197 197 197 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 124 123 124 211 208 211 T T T <--- 18 360-379 124 116 124 209 196 209 F T F <--- 19 380-399 120 120 120 203 203 203 F F F 20 400-419 122 122 122 205 205 205 F F F 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 122 122 122 205 205 205 F F F 24 480-499 122 122 122 206 206 206 F F F 25 500-519 124 100 124 211 168 211 F F F <--- 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 119 124 210 202 210 F F F <--- 28 560-579 124 122 124 209 205 209 F F F <--- 29 580-599 112 112 112 186 186 186 F F F Max gen: 124, Max flush: 211 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 196 196 196 F F F 1 020-039 125 125 125 212 212 212 F F F 2 040-059 115 115 115 195 195 195 F F F 3 060-079 122 122 122 205 205 205 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 125 125 125 212 212 212 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 125 125 125 212 212 212 F F F 9 180-199 125 125 125 212 212 212 F F F 10 200-219 124 124 124 209 209 209 F F F 11 220-239 124 124 124 209 209 209 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 124 124 124 209 209 209 F F F 15 300-319 118 118 118 197 197 197 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 125 125 125 212 212 212 F F F 18 360-379 124 124 124 209 209 209 F F F 19 380-399 120 120 120 203 203 203 F F F 20 400-419 122 122 122 205 205 205 F F F 21 420-439 119 119 119 202 202 202 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 122 122 122 205 205 205 F F F 24 480-499 122 122 122 206 206 206 F F F 25 500-519 124 124 124 211 211 211 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 124 124 210 210 210 F F F 28 560-579 124 124 124 209 209 209 F F F 29 580-599 112 112 112 186 186 186 F F F Max gen: 125, Max flush: 212 Begin loop 63 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.602655299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.614276208Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.61458781Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.614626588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.614638943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.614656881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"Crucible 6883edb7-0485-4dd6-af2f-424b2c4768e6 has session id: c3998b1b-362c-4295-91e8-ea6b92f9728b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615063667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Upstairs opts: Upstairs UUID: 6883edb7-0485-4dd6-af2f-424b2c4768e6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615101544Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Crucible stats registered with UUID: 6883edb7-0485-4dd6-af2f-424b2c4768e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615117491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:15.615131057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615293701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615326376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615342003Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615475094Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615491991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"6883edb7-0485-4dd6-af2f-424b2c4768e6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615522775Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615536602Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615550508Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615564534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615578171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615592357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615662159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"ds_connection connected from Ok(127.0.0.1:60295)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615692493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"ds_connection connected from Ok(127.0.0.1:54639)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.61570878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43244)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.615725858Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616289594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616348571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616377394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616701112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616745672Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[0]R flush_numbers[0..12]: [196, 212, 195, 205, 212, 212, 212, 210, 212, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616817605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[0]R generation[0..12]: [116, 125, 115, 122, 125, 125, 125, 124, 125, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616834803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.6168512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[1]R flush_numbers[0..12]: [196, 212, 195, 205, 212, 212, 212, 210, 212, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616867467Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[1]R generation[0..12]: [116, 125, 115, 122, 125, 125, 125, 124, 125, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616883335Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616899882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[2]R flush_numbers[0..12]: [196, 212, 195, 205, 212, 212, 212, 210, 212, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616918471Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[2]R generation[0..12]: [116, 125, 115, 122, 125, 125, 125, 124, 125, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616934248Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616950415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616964081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.61698376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Next flush: 213","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.616997217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617010943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"":"downstairs","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617025259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617037765Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"0","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617051711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"1","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617065868Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"client":"2","":"downstairs","session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617082445Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617096132Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"6883edb7-0485-4dd6-af2f-424b2c4768e6 is now active with session: c3998b1b-362c-4295-91e8-ea6b92f9728b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617110948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617123634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712,"session_id":"c3998b1b-362c-4295-91e8-ea6b92f9728b"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617139561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} {"msg":"Activated sub_volume 6883edb7-0485-4dd6-af2f-424b2c4768e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:15.617151007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 584 len 8192 data: 14 12 02/30 Write block 377 len 36864 data: 10 12 14 15 14 17 19 16 13 03/30 Write block 282 len 4096 data: 13 04/30 Read block 560 len 36864 05/30 Write block 7 len 28672 data: 7 10 10 9 8 9 9 06/30 Write block 173 len 20480 data: 14 12 9 8 10 07/30 Write block 358 len 12288 data: 12 11 10 08/30 Write block 466 len 8192 data: 9 8 09/30 Write block 37 len 16384 data: 7 8 7 7 10/30 Read block 19 len 24576 11/30 Read block 498 len 40960 12/30 Read block 27 len 12288 13/30 Flush 14/30 Write block 409 len 16384 data: 12 11 12 13 15/30 Read block 520 len 28672 16/30 Write block 302 len 36864 data: 7 7 4 6 7 10 11 9 9 17/30 Read block 492 len 32768 18/30 Read block 166 len 40960 19/30 Read block 346 len 8192 20/30 Write block 433 len 28672 data: 12 7 9 13 14 16 16 21/30 Read block 193 len 12288 22/30 Read block 489 len 20480 23/30 Write block 425 len 12288 data: 14 13 12 24/30 Write block 167 len 12288 data: 11 11 12 25/30 Write block 497 len 36864 data: 11 13 12 12 11 9 11 11 13 26/30 Read block 113 len 36864 27/30 Read block 162 len 16384 28/30 Read block 553 len 4096 29/30 Read block 305 len 20480 30/30 Read block 488 len 4096 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Sent false 1001 Acked Write 9 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1012)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:15.628664177Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1712} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 116 213 213 196 F F F <--- 1 020-039 126 126 125 213 213 212 F F F <--- 2 040-059 126 126 115 213 213 195 F F F <--- 3 060-079 122 122 122 205 205 205 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 125 125 125 212 212 212 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 126 126 125 213 213 212 T T F <--- 9 180-199 125 125 125 212 212 212 F F F 10 200-219 124 124 124 209 209 209 F F F 11 220-239 124 124 124 209 209 209 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 126 126 124 213 213 209 F F F <--- 15 300-319 118 118 118 197 197 197 T T F <--- 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 125 213 213 212 F F F <--- 18 360-379 126 126 124 213 213 209 F F F <--- 19 380-399 126 126 120 213 213 203 F F F <--- 20 400-419 122 122 122 205 205 205 T T F <--- 21 420-439 119 119 119 202 202 202 T T F <--- 22 440-459 119 119 119 202 202 202 F F F 23 460-479 126 126 122 213 213 205 F F F <--- 24 480-499 122 122 122 206 206 206 T T F <--- 25 500-519 124 124 124 211 211 211 T T F <--- 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 124 124 210 210 210 F F F 28 560-579 124 124 124 209 209 209 F F F 29 580-599 126 126 112 213 213 186 F F F <--- Max gen: 126, Max flush: 213 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 213 213 213 F F F 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 F F F 3 060-079 122 122 122 205 205 205 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 125 125 125 212 212 212 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 127 127 127 214 214 214 F F F 9 180-199 125 125 125 212 212 212 F F F 10 200-219 124 124 124 209 209 209 F F F 11 220-239 124 124 124 209 209 209 F F F 12 240-259 120 120 120 203 203 203 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 126 126 126 213 213 213 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 126 126 126 213 213 213 F F F 19 380-399 126 126 126 213 213 213 F F F 20 400-419 127 127 127 214 214 214 F F F 21 420-439 127 127 127 214 214 214 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 126 126 126 213 213 213 F F F 24 480-499 127 127 127 214 214 214 F F F 25 500-519 127 127 127 214 214 214 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 124 124 210 210 210 F F F 28 560-579 124 124 124 209 209 209 F F F 29 580-599 126 126 126 213 213 213 F F F Max gen: 127, Max flush: 214 Begin loop 64 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.963995239Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.975651454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.975969949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976009657Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976021872Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.97603954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"Crucible 91b1b8b7-af22-4e63-aca9-0f2494d19fa5 has session id: 042152e6-7f30-4795-9892-43ef0d2a43c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.9764125Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Upstairs opts: Upstairs UUID: 91b1b8b7-af22-4e63-aca9-0f2494d19fa5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976447416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Crucible stats registered with UUID: 91b1b8b7-af22-4e63-aca9-0f2494d19fa5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976463323Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:19.976476739Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976628488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976652169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976666956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976798776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976815664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"91b1b8b7-af22-4e63-aca9-0f2494d19fa5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976830911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976844017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976858183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976872459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976886446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976900962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.976915309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:42700)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.977006711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:62352)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.977035484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:51155)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.977051221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.9777296Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.977757742Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.977778482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978088163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978132123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[0]R flush_numbers[0..12]: [213, 213, 213, 205, 212, 212, 212, 210, 214, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978195542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[0]R generation[0..12]: [126, 126, 126, 122, 125, 125, 125, 124, 127, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978212779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978229377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[1]R flush_numbers[0..12]: [213, 213, 213, 205, 212, 212, 212, 210, 214, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978244994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[1]R generation[0..12]: [126, 126, 126, 122, 125, 125, 125, 124, 127, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978260901Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978277299Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[2]R flush_numbers[0..12]: [213, 213, 213, 205, 212, 212, 212, 210, 214, 212, 209, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978292876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[2]R generation[0..12]: [126, 126, 126, 122, 125, 125, 125, 124, 127, 125, 124, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978308643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.97832483Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978338357Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":Wait for a query_work_queue command to finish before sending IO "downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978359266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Next flush: 215","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978374553Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978390861Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"":"downstairs","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978404887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978417533Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"0","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978432069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"1","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978446226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"client":"2","":"downstairs","session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978462553Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978476269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"91b1b8b7-af22-4e63-aca9-0f2494d19fa5 is now active with session: 042152e6-7f30-4795-9892-43ef0d2a43c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978491326Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978504552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720,"session_id":"042152e6-7f30-4795-9892-43ef0d2a43c9"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978517808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} {"msg":"Activated sub_volume 91b1b8b7-af22-4e63-aca9-0f2494d19fa5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:19.978529714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 245 len 40960 data: 10 13 14 14 15 13 10 12 10 14 02/30 Read block 30 len 8192 03/30 Read block 452 len 20480 04/30 Read block 388 len 8192 05/30 Write block 375 len 16384 data: 11 9 11 13 06/30 Read block 6 len 40960 07/30 Flush 08/30 Write block 377 len 40960 data: 12 14 15 16 15 18 20 17 14 14 09/30 Write block 131 len 20480 data: 19 16 17 16 16 10/30 Read block 165 len 4096 11/30 Flush 12/30 Write block 74 len 20480 data: 10 13 12 13 12 13/30 Read block 279 len 12288 14/30 Write block 508 len 20480 data: 10 10 11 10 12 15/30 Write block 393 len 28672 data: 12 12 9 11 14 13 13 16/30 Read block 397 len 8192 17/30 Read block 540 len 4096 18/30 Read block 86 len 28672 19/30 Write block 195 len 40960 data: 13 12 12 9 9 9 6 6 5 4 20/30 Read block 482 len 40960 21/30 Write block 497 len 8192 data: 12 14 22/30 Write block 593 len 24576 data: 9 9 8 8 8 5 23/30 Flush 24/30 Flush 25/30 Write block 379 len 40960 data: 16 17 16 19 21 18 15 15 15 14 26/30 Flush 27/30 Write block 293 len 4096 data: 11 28/30 Read block 469 len 20480 29/30 Read block 115 len 4096 30/30 Read block 566 len 8192 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 2 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Read 10 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 1 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:19.991102406Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1720} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 213 213 213 F F F 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 F F F 3 060-079 128 128 122 217 217 205 F F F <--- 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 128 128 125 216 216 212 F F F <--- 7 140-159 124 124 124 210 210 210 F F F 8 160-179 127 127 127 214 214 214 F F F 9 180-199 128 128 125 217 217 212 F F F <--- 10 200-219 128 128 124 217 217 209 F F F <--- 11 220-239 124 124 124 209 209 209 F F F 12 240-259 128 128 120 215 215 203 F F T <--- 13 260-279 124 124 124 209 209 209 F F F 14 280-299 126 126 126 213 213 213 T T F <--- 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 128 128 126 219 219 213 F F F <--- 19 380-399 128 128 126 219 219 213 F F F <--- 20 400-419 127 127 127 214 214 214 F F F 21 420-439 127 127 127 214 214 214 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 126 126 126 213 213 213 F F F 24 480-499 128 128 127 217 217 214 F F F <--- 25 500-519 128 128 127 217 217 214 F F F <--- 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 124 124 210 210 210 F F F 28 560-579 124 124 124 209 209 209 F F F 29 580-599 128 128 126 217 217 213 F F F <--- Max gen: 128, Max flush: 219 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 213 213 213 F F F 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 F F F 3 060-079 128 128 128 217 217 217 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 128 128 128 216 216 216 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 127 127 127 214 214 214 F F F 9 180-199 128 128 128 217 217 217 F F F 10 200-219 128 128 128 217 217 217 F F F 11 220-239 124 124 124 209 209 209 F F F 12 240-259 128 128 128 215 215 215 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 128 128 128 219 219 219 F F F 19 380-399 128 128 128 219 219 219 F F F 20 400-419 127 127 127 214 214 214 F F F 21 420-439 127 127 127 214 214 214 F F F 22 440-459 119 119 119 202 202 202 F F F 23 460-479 126 126 126 213 213 213 F F F 24 480-499 128 128 128 217 217 217 F F F 25 500-519 128 128 128 217 217 217 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 124 124 124 210 210 210 F F F 28 560-579 124 124 124 209 209 209 F F F 29 580-599 128 128 128 217 217 217 F F F Max gen: 129, Max flush: 220 Begin loop 65 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.312388876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324195169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.32454721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324585837Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324598123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324616181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"Crucible 3fc9287d-0969-4120-8666-85d0330f607d has session id: 00a69a20-698b-44e3-ad53-99b9aced3cf3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324950243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Upstairs opts: Upstairs UUID: 3fc9287d-0969-4120-8666-85d0330f607d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.324984869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Crucible stats registered with UUID: 3fc9287d-0969-4120-8666-85d0330f607d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325000266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:24.325013342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325181368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325216134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325232061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325330646Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325346153Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"3fc9287d-0969-4120-8666-85d0330f607d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325367493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325380089Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325394225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325408262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325421978Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325436304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325452722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:46182)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325500524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47503)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325622209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:64123)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.325641178Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326269403Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326297536Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326314374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.32663575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326684062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[0]R flush_numbers[0..12]: [213, 213, 213, 217, 212, 212, 216, 210, 214, 217, 217, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326719878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[0]R generation[0..12]: [126, 126, 126, 128, 125, 125, 128, 124, 127, 128, 128, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326737016Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326753874Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[1]R flush_numbers[0..12]: [213, 213, 213, 217, 212, 212, 216, 210, 214, 217, 217, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326769511Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[1]R generation[0..12]: [126, 126, 126, 128, 125, 125, 128, 124, 127, 128, 128, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326785358Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326804477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[2]R flush_numbers[0..12]: [213, 213, 213, 217, 212, 212, 216, 210, 214, 217, 217, 209]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326820304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[2]R generation[0..12]: [126, 126, 126, 128, 125, 125, 128, 124, 127, 128, 128, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326835751Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326852109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326865665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326879911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Next flush: 221","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326893137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326906333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"":"downstairs","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Wait for a query_work_queue command to finish before sending IO No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326924592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326937508Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"0","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326951634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"1","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.32696554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"client":"2","":"downstairs","session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326979717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.326993073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"3fc9287d-0969-4120-8666-85d0330f607d is now active with session: 00a69a20-698b-44e3-ad53-99b9aced3cf3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.32700803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"Set Active after no reconciliationDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.327025227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728,"session_id":"00a69a20-698b-44e3-ad53-99b9aced3cf3"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.327039204Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} {"msg":"Activated sub_volume 3fc9287d-0969-4120-8666-85d0330f607d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:24.327050759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 571 len 12288 data: 12 12 9 02/30 Read block 22 len 36864 03/30 Write block 244 len 20480 data: 9 11 14 15 15 04/30 Write block 227 len 4096 data: 14 05/30 Flush 06/30 Write block 502 len 24576 data: 10 12 12 14 11 11 07/30 Write block 546 len 12288 data: 11 10 12 08/30 Flush 09/30 Flush 10/30 Write block 445 len 40960 data: 9 7 5 9 10 10 9 10 9 7 11/30 Write block 175 len 16384 data: 10 9 11 10 12/30 Read block 485 len 24576 13/30 Read block 389 len 16384 14/30 Write block 430 len 28672 data: 13 13 14 13 8 10 14 15/30 Read block 494 len 24576 16/30 Write block 217 len 24576 data: 12 10 10 14 12 10 17/30 Write block 430 len 24576 data: 14 14 15 14 9 11 18/30 Write block 186 len 24576 data: 11 12 12 16 13 15 19/30 Write block 181 len 28672 data: 8 9 9 10 12 12 13 20/30 Read block 34 len 32768 21/30 Flush 22/30 Write block 64 len 24576 data: 9 9 10 10 11 12 23/30 Read block 275 len 40960 24/30 Read block 460 len 28672 25/30 Write block 365 len 28672 data: 10 7 8 11 13 12 9 26/30 Flush 27/30 Write block 3 len 4096 data: 7 28/30 Write block 253 len 24576 data: 11 15 13 12 11 10 29/30 Read block 141 len 8192 30/30 Read block 214 len 32768 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:24.339592437Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1728} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 213 213 213 T F T <--- 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 F F F 3 060-079 130 128 130 225 217 225 F F F <--- 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 128 128 128 216 216 216 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 130 127 130 224 214 224 F F F <--- 9 180-199 130 128 130 224 217 224 F F F <--- 10 200-219 130 128 130 224 217 224 F F F <--- 11 220-239 130 124 130 224 209 224 F F F <--- 12 240-259 130 128 130 221 215 221 T T T <--- 13 260-279 124 124 124 209 209 209 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 130 128 130 225 219 225 F F F <--- 19 380-399 128 128 128 219 219 219 F F F 20 400-419 127 127 127 214 214 214 F F F 21 420-439 130 127 130 224 214 224 F F F <--- 22 440-459 130 119 130 224 202 224 F F F <--- 23 460-479 126 126 126 213 213 213 F F F 24 480-499 128 128 128 217 217 217 F F F 25 500-519 130 128 130 222 217 222 F F F <--- 26 520-539 123 123 123 208 208 208 F F F 27 540-559 130 124 130 222 210 222 F F F <--- 28 560-579 130 124 130 221 209 221 F T F <--- 29 580-599 128 128 128 217 217 217 F F F Max gen: 130, Max flush: 225 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 226 226 226 F F F 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 F F F 3 060-079 130 130 130 225 225 225 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 125 125 125 212 212 212 F F F 6 120-139 128 128 128 216 216 216 F F F 7 140-159 124 124 124 210 210 210 F F F 8 160-179 130 130 130 224 224 224 F F F 9 180-199 130 130 130 224 224 224 F F F 10 200-219 130 130 130 224 224 224 F F F 11 220-239 130 130 130 224 224 224 F F F 12 240-259 131 131 131 226 226 226 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 130 130 130 225 225 225 F F F 19 380-399 128 128 128 219 219 219 F F F 20 400-419 127 127 127 214 214 214 F F F 21 420-439 130 130 130 224 224 224 F F F 22 440-459 130 130 130 224 224 224 F F F 23 460-479 126 126 126 213 213 213 F F F 24 480-499 128 128 128 217 217 217 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 130 130 130 222 222 222 F F F 28 560-579 130 130 130 221 221 221 F F F 29 580-599 128 128 128 217 217 217 F F F Max gen: 131, Max flush: 226 Begin loop 66 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.664915253Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.676678547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677004705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677051016Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677063572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.67708149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"Crucible 398c05ba-5a8b-4a97-ad9a-74013fb2be21 has session id: 2d9cac78-2774-49cc-b0c7-cd72409b0fd5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677424977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Upstairs opts: Upstairs UUID: 398c05ba-5a8b-4a97-ad9a-74013fb2be21, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677459462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Crucible stats registered with UUID: 398c05ba-5a8b-4a97-ad9a-74013fb2be21","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.6774754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:28.677489156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677635983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677662725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677677632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.6778058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677823418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"398c05ba-5a8b-4a97-ad9a-74013fb2be21 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677853472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677867148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677881444Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677895601Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.67791511Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677929546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.677943853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37844)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678034404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:48796)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678158571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:43163)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678177159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678753801Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678807566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.678828125Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679131013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679175373Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[0]R flush_numbers[0..12]: [226, 213, 213, 225, 212, 212, 216, 210, 224, 224, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679247596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[0]R generation[0..12]: [131, 126, 126, 130, 125, 125, 128, 124, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679264484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679281091Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[1]R flush_numbers[0..12]: [226, 213, 213, 225, 212, 212, 216, 210, 224, 224, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679296929Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[1]R generation[0..12]: [131, 126, 126, 130, 125, 125, 128, 124, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679312536Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679328893Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[2]R flush_numbers[0..12]: [226, 213, 213, 225, 212, 212, 216, 210, 224, 224, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.67934436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[2]R generation[0..12]: [131, 126, 126, 130, 125, 125, 128, 124, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679359697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679376215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679389381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.67940887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Next flush: 227","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679424157Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679437563Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"":"downstairs","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679451569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679464055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"0","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679477931Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"1","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679491447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"client":"2","":"downstairs","session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:05:28.679505644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679521161Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"398c05ba-5a8b-4a97-ad9a-74013fb2be21 is now active with session: 2d9cac78-2774-49cc-b0c7-cd72409b0fd5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679536017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679548613Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736,"session_id":"2d9cac78-2774-49cc-b0c7-cd72409b0fd5"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679562019Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} {"msg":"Activated sub_volume 398c05ba-5a8b-4a97-ad9a-74013fb2be21","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:28.679573505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 391 len 16384 data: 12 12 13 13 02/30 Write block 409 len 20480 data: 13 12 13 14 12 03/30 Read block 575 len 12288 04/30 Write block 142 len 40960 data: 9 10 11 9 9 8 10 11 12 13 05/30 Write block 12 len 16384 data: 10 10 6 6 06/30 Write block 116 len 40960 data: 8 8 8 9 9 10 10 10 12 13 07/30 Read block 100 len 16384 08/30 Read block 137 len 20480 09/30 Write block 0 len 24576 data: 4 4 5 8 7 10 10/30 Read block 325 len 8192 11/30 Read block 393 len 20480 12/30 Read block 534 len 8192 13/30 Read block 447 len 20480 14/30 Read block 470 len 28672 15/30 Write block 366 len 28672 data: 8 9 12 14 13 10 11 16/30 Write block 186 len 32768 data: 13 14 13 17 14 16 12 9 17/30 Read block 120 len 24576 18/30 Write block 112 len 40960 data: 10 9 7 8 9 9 9 10 10 11 19/30 Flush 20/30 Flush 21/30 Write block 51 len 24576 data: 14 12 15 14 12 10 22/30 Write block 479 len 20480 data: 14 14 14 12 10 23/30 Read block 252 len 32768 24/30 Read block 583 len 36864 25/30 Write block 475 len 16384 data: 10 8 11 12 26/30 Write block 58 len 28672 data: 9 9 9 7 8 8 10 27/30 Write block 431 len 40960 data: 15 16 15 10 12 15 15 17 17 14 28/30 Read block 130 len 16384 29/30 Read block 377 len 28672 30/30 Read block 207 len 4096 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Write 5 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:28.692159013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1736} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 131 132 227 226 227 F F F <--- 1 020-039 126 126 126 213 213 213 F F F 2 040-059 126 126 126 213 213 213 T F T <--- 3 060-079 130 130 130 225 225 225 T F T <--- 4 080-099 125 125 125 212 212 212 F F F 5 100-119 132 125 132 227 212 227 F F F <--- 6 120-139 132 128 132 227 216 227 F F F <--- 7 140-159 132 124 132 227 210 227 F F F <--- 8 160-179 130 130 130 224 224 224 F F F 9 180-199 132 130 132 227 224 227 F F F <--- 10 200-219 130 130 130 224 224 224 F F F 11 220-239 130 130 130 224 224 224 F F F 12 240-259 131 131 131 226 226 226 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 132 130 132 227 225 227 F F F <--- 19 380-399 132 128 132 227 219 227 F F F <--- 20 400-419 132 127 132 227 214 227 F F F <--- 21 420-439 130 130 130 224 224 224 T F T <--- 22 440-459 130 130 130 224 224 224 T F T <--- 23 460-479 126 126 126 213 213 213 T F T <--- 24 480-499 128 128 128 217 217 217 T F T <--- 25 500-519 130 130 130 222 222 222 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 130 130 130 222 222 222 F F F 28 560-579 130 130 130 221 221 221 F F F 29 580-599 128 128 128 217 217 217 F F F Max gen: 132, Max flush: 227 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 227 227 227 F F F 1 020-039 126 126 126 213 213 213 F F F 2 040-059 133 133 133 228 228 228 F F F 3 060-079 133 133 133 228 228 228 F F F 4 080-099 125 125 125 212 212 212 F F F 5 100-119 132 132 132 227 227 227 F F F 6 120-139 132 132 132 227 227 227 F F F 7 140-159 132 132 132 227 227 227 F F F 8 160-179 130 130 130 224 224 224 F F F 9 180-199 132 132 132 227 227 227 F F F 10 200-219 130 130 130 224 224 224 F F F 11 220-239 130 130 130 224 224 224 F F F 12 240-259 131 131 131 226 226 226 F F F 13 260-279 124 124 124 209 209 209 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F F F 18 360-379 132 132 132 227 227 227 F F F 19 380-399 132 132 132 227 227 227 F F F 20 400-419 132 132 132 227 227 227 F F F 21 420-439 133 133 133 228 228 228 F F F 22 440-459 133 133 133 228 228 228 F F F 23 460-479 133 133 133 228 228 228 F F F 24 480-499 133 133 133 228 228 228 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 123 123 123 208 208 208 F F F 27 540-559 130 130 130 222 222 222 F F F 28 560-579 130 130 130 221 221 221 F F F 29 580-599 128 128 128 217 217 217 F F F Max gen: 133, Max flush: 228 Begin loop 67 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.022245506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:33.032894133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.032948097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.043359966Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.043649138Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.043688015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.043700381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.043718219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"Crucible 97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4 has session id: 33df644b-e74d-48c9-883b-bd6b79ba7504","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044100843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Upstairs opts: Upstairs UUID: 97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044136469Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Crucible stats registered with UUID: 97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044152337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:33.044165593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044311389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044346435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044362212Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044477375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044492021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044509859Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044522755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044539983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.04455428Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044583243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044617188Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044638898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"ds_connection connected from Ok(127.0.0.1:36143)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044654225Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"ds_connection connected from Ok(127.0.0.1:64561)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044669232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"ds_connection connected from Ok(127.0.0.1:38095)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.044771598Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045326131Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045384587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045536677Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045682183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.04572026Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[0]R flush_numbers[0..12]: [227, 213, 228, 228, 212, 227, 227, 227, 224, 227, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045943082Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[0]R generation[0..12]: [132, 126, 133, 133, 125, 132, 132, 132, 130, 132, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045970774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.045991254Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[1]R flush_numbers[0..12]: [227, 213, 228, 228, 212, 227, 227, 227, 224, 227, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046007651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[1]R generation[0..12]: [132, 126, 133, 133, 125, 132, 132, 132, 130, 132, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046023228Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046039305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[2]R flush_numbers[0..12]: [227, 213, 228, 228, 212, 227, 227, 227, 224, 227, 224, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046055193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[2]R generation[0..12]: [132, 126, 133, 133, 125, 132, 132, 132, 130, 132, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.0460708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046087307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:05:33.046100983Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046121293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Next flush: 229","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046134759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046148045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"":"downstairs","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046162231Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046174547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"0","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046188553Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"1","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046206051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"client":"2","":"downstairs","session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046220998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046234324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4 is now active with session: 33df644b-e74d-48c9-883b-bd6b79ba7504","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046249111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046262027Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744,"session_id":"33df644b-e74d-48c9-883b-bd6b79ba7504"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046275243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} {"msg":"Activated sub_volume 97f2cf58-9e4d-497e-a8b6-31ddbba2d5d4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:33.046286788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 479 len 12288 data: 15 15 15 02/30 Write block 392 len 20480 data: 13 14 14 10 12 03/30 Write block 165 len 40960 data: 8 10 12 12 13 14 15 16 15 13 04/30 Read block 388 len 4096 05/30 Read block 412 len 36864 06/30 Write block 253 len 32768 data: 12 16 14 13 12 11 9 9 07/30 Read block 273 len 40960 08/30 Write block 535 len 28672 data: 10 7 7 10 14 13 13 09/30 Write block 371 len 4096 data: 11 10/30 Read block 473 len 32768 11/30 Write block 122 len 12288 data: 11 11 13 12/30 Write block 453 len 8192 data: 10 8 13/30 Read block 146 len 20480 14/30 Read block 129 len 4096 15/30 Write block 91 len 28672 data: 17 15 13 10 9 9 9 16/30 Write block 577 len 40960 data: 12 12 12 15 15 13 13 15 13 10 17/30 Flush 18/30 Write block 124 len 32768 data: 14 14 15 18 17 16 15 20 19/30 Write block 195 len 28672 data: 14 13 13 10 10 10 7 20/30 Read block 31 len 4096 21/30 Write block 355 len 36864 data: 11 11 11 13 12 11 7 8 8 22/30 Read block 11 len 36864 23/30 Read block 446 len 20480 24/30 Write block 35 len 4096 data: 9 25/30 Write block 422 len 8192 data: 11 12 26/30 Write block 8 len 12288 data: 11 11 10 27/30 Write block 473 len 12288 data: 10 8 11 28/30 Read block 102 len 20480 29/30 Read block 294 len 36864 30/30 Write block 496 len 16384 data: 10 13 15 13 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:33.058157421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1744} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 227 227 227 F T T <--- 1 020-039 126 126 126 213 213 213 F T T <--- 2 040-059 133 133 133 228 228 228 F F F 3 060-079 133 133 133 228 228 228 F F F 4 080-099 125 134 134 212 229 229 F F F <--- 5 100-119 132 132 132 227 227 227 F F F 6 120-139 132 134 134 227 229 229 F T T <--- 7 140-159 132 132 132 227 227 227 F F F 8 160-179 130 134 134 224 229 229 F F F <--- 9 180-199 132 132 132 227 227 227 F T T <--- 10 200-219 130 130 130 224 224 224 F T T <--- 11 220-239 130 130 130 224 224 224 F F F 12 240-259 131 134 134 226 229 229 F F F <--- 13 260-279 124 134 134 209 229 229 F F F <--- 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 126 126 126 213 213 213 F T T <--- 18 360-379 132 134 134 227 229 229 F T T <--- 19 380-399 132 134 134 227 229 229 F F F <--- 20 400-419 132 132 132 227 227 227 F F F 21 420-439 133 133 133 228 228 228 F T T <--- 22 440-459 133 134 134 228 229 229 F F F <--- 23 460-479 133 134 134 228 229 229 F T T <--- 24 480-499 133 134 134 228 229 229 F T T <--- 25 500-519 130 130 130 222 222 222 F F F 26 520-539 123 134 134 208 229 229 F F F <--- 27 540-559 130 134 134 222 229 229 F F F <--- 28 560-579 130 134 134 221 229 229 F F F <--- 29 580-599 128 134 134 217 229 229 F F F <--- Max gen: 134, Max flush: 229 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 135 135 135 230 230 230 F F F 2 040-059 133 133 133 228 228 228 F F F 3 060-079 133 133 133 228 228 228 F F F 4 080-099 134 134 134 229 229 229 F F F 5 100-119 132 132 132 227 227 227 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 132 132 132 227 227 227 F F F 8 160-179 134 134 134 229 229 229 F F F 9 180-199 135 135 135 230 230 230 F F F 10 200-219 135 135 135 230 230 230 F F F 11 220-239 130 130 130 224 224 224 F F F 12 240-259 134 134 134 229 229 229 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 129 129 129 220 220 220 F F F 15 300-319 127 127 127 214 214 214 F F F 16 320-339 122 122 122 205 205 205 F F F 17 340-359 135 135 135 230 230 230 F F F 18 360-379 135 135 135 230 230 230 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 132 132 132 227 227 227 F F F 21 420-439 135 135 135 230 230 230 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 135 135 135 230 230 230 F F F 24 480-499 135 135 135 230 230 230 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 134 134 134 229 229 229 F F F 27 540-559 134 134 134 229 229 229 F F F 28 560-579 134 134 134 229 229 229 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 135, Max flush: 230 Begin loop 68 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.415537477Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.427391552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.427703724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.427742742Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.427754847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.427772755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"Crucible 7db8adbc-2058-4fa7-9fdc-cf210d901ba4 has session id: 74c54945-7f8c-4181-bb7d-bea1e439b70f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428141323Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Upstairs opts: Upstairs UUID: 7db8adbc-2058-4fa7-9fdc-cf210d901ba4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.42817796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Crucible stats registered with UUID: 7db8adbc-2058-4fa7-9fdc-cf210d901ba4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428198029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:37.428211745Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428360423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428395429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428410886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428516114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428531661Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"7db8adbc-2058-4fa7-9fdc-cf210d901ba4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.42855165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428564476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428578452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428591798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428605624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428619911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428634047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55330)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428668013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"ds_connection connected from Ok(127.0.0.1:56490)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428686781Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55969)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.428701988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429456322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429494449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429511487Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429839926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429884196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[0]R flush_numbers[0..12]: [230, 230, 228, 228, 229, 227, 230, 227, 229, 230, 230, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429921763Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[0]R generation[0..12]: [135, 135, 133, 133, 134, 132, 135, 132, 134, 135, 135, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429938621Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429955099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[1]R flush_numbers[0..12]: [230, 230, 228, 228, 229, 227, 230, 227, 229, 230, 230, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429970566Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[1]R generation[0..12]: [135, 135, 133, 133, 134, 132, 135, 132, 134, 135, 135, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.429986613Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.43000315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[2]R flush_numbers[0..12]: [230, 230, 228, 228, 229, 227, 230, 227, 229, 230, 230, 224]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430018998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[2]R generation[0..12]: [135, 135, 133, 133, 134, 132, 135, 132, 134, 135, 135, 130]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430037796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430054224Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":Wait for a query_work_queue command to finish before sending IO "2026-03-14T05:05:37.430082637Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Next flush: 231","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430102106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430115532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"":"downstairs","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430129598Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430142224Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"0","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.4301561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"1","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430170096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"client":"2","":"downstairs","session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430184413Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.43020033Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"7db8adbc-2058-4fa7-9fdc-cf210d901ba4 is now active with session: 74c54945-7f8c-4181-bb7d-bea1e439b70f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430215087Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430227623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752,"session_id":"74c54945-7f8c-4181-bb7d-bea1e439b70f"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430241179Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} {"msg":"Activated sub_volume 7db8adbc-2058-4fa7-9fdc-cf210d901ba4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:37.430252914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 581 len 12288 data: 16 14 14 03/30 Read block 563 len 12288 04/30 Write block 349 len 40960 data: 12 10 12 11 10 11 12 12 12 14 05/30 Write block 340 len 28672 data: 13 13 8 10 9 9 9 06/30 Read block 316 len 28672 07/30 Write block 575 len 32768 data: 14 10 13 13 13 16 17 15 08/30 Read block 332 len 20480 09/30 Write block 477 len 24576 data: 12 13 16 16 16 13 10/30 Write block 57 len 4096 data: 8 11/30 Flush 12/30 Flush 13/30 Write block 318 len 40960 data: 16 17 18 15 14 13 12 11 14 11 14/30 Read block 331 len 16384 15/30 Read block 153 len 4096 16/30 Write block 533 len 16384 data: 10 9 11 8 17/30 Write block 97 len 24576 data: 10 8 7 8 8 7 18/30 Write block 317 len 36864 data: 15 17 18 19 16 15 14 13 12 19/30 Read block 237 len 4096 20/30 Write block 444 len 28672 data: 8 10 8 6 10 11 11 21/30 Write block 579 len 8192 data: 14 17 22/30 Write block 290 len 20480 data: 14 13 12 12 11 23/30 Flush 24/30 Write block 288 len 24576 data: 14 17 15 14 13 13 25/30 Write block 249 len 28672 data: 16 14 11 13 13 17 15 26/30 Write block 66 len 36864 data: 11 11 12 13 9 8 8 9 11 27/30 Write block 490 len 8192 data: 13 12 28/30 Read block 224 len 28672 29/30 Write block 225 len 20480 data: 13 14 15 13 14 30/30 Write block 175 len 24576 data: 11 10 12 11 7 8 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 5 Done Done Done false 1008 Acked Write 6 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 22 26 Done 27 27 7 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:37.44246006Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1752} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 135 135 135 230 230 230 F F F 2 040-059 136 136 133 232 232 228 F F F <--- 3 060-079 133 133 133 228 228 228 T T F <--- 4 080-099 136 136 134 234 234 229 F F F <--- 5 100-119 136 136 132 234 234 227 F F F <--- 6 120-139 135 135 135 230 230 230 F F F 7 140-159 132 132 132 227 227 227 F F F 8 160-179 134 134 134 229 229 229 T T F <--- 9 180-199 135 135 135 230 230 230 T T F <--- 10 200-219 135 135 135 230 230 230 F F F 11 220-239 130 130 130 224 224 224 T T F <--- 12 240-259 134 134 134 229 229 229 T T F <--- 13 260-279 134 134 134 229 229 229 F F F 14 280-299 136 136 129 234 234 220 T T F <--- 15 300-319 136 136 127 234 234 214 F F F <--- 16 320-339 136 136 122 234 234 205 F F F <--- 17 340-359 136 136 135 232 232 230 F F T <--- 18 360-379 135 135 135 230 230 230 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 132 132 132 227 227 227 F F F 21 420-439 135 135 135 230 230 230 F F F 22 440-459 136 136 134 234 234 229 F F F <--- 23 460-479 136 136 135 232 232 230 F F F <--- 24 480-499 136 136 135 232 232 230 T T F <--- 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 134 234 234 229 F F F <--- 27 540-559 134 134 134 229 229 229 F F F 28 560-579 136 136 134 234 234 229 F F T <--- 29 580-599 136 136 134 234 234 229 F F T <--- Max gen: 136, Max flush: 234 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 135 135 135 230 230 230 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 132 132 132 227 227 227 F F F 8 160-179 137 137 137 235 235 235 F F F 9 180-199 137 137 137 235 235 235 F F F 10 200-219 135 135 135 230 230 230 F F F 11 220-239 137 137 137 235 235 235 F F F 12 240-259 137 137 137 235 235 235 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 136 136 136 234 234 234 F F F 16 320-339 136 136 136 234 234 234 F F F 17 340-359 136 136 136 232 232 232 F F F 18 360-379 135 135 135 230 230 230 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 132 132 132 227 227 227 F F F 21 420-439 135 135 135 230 230 230 F F F 22 440-459 136 136 136 234 234 234 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 134 134 134 229 229 229 F F F 28 560-579 136 136 136 234 234 234 F F F 29 580-599 136 136 136 234 234 234 F F F Max gen: 137, Max flush: 235 Begin loop 69 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.786135349Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.797825289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798196168Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798233495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798246361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798264059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"Crucible d467e58b-6506-4bf3-ad93-75a08b01d25b has session id: 68d2db95-f216-4a3d-ab0f-9e1273180f04","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.7986395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Upstairs opts: Upstairs UUID: d467e58b-6506-4bf3-ad93-75a08b01d25b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798674036Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Crucible stats registered with UUID: d467e58b-6506-4bf3-ad93-75a08b01d25b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798690343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:41.798703999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798857059Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798891615Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.798907332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799024735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799040332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"d467e58b-6506-4bf3-ad93-75a08b01d25b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799070466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799084202Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799098429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799112095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"2","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799151733Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799175724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799191051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"ds_connection connected from Ok(127.0.0.1:50912)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.79929964Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"ds_connection connected from Ok(127.0.0.1:65373)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799407139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"ds_connection connected from Ok(127.0.0.1:34731)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.799434702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800049992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800069621Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"2","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800096283Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800413687Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800456367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[0]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 227, 235, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800516924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[0]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 132, 137, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800534062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.80055069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[1]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 227, 235, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800566617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[1]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 132, 137, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800581984Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800598371Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[2]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 227, 235, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800614169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[2]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 132, 137, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800629616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800645813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800659509Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800679798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Next flush: 236","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800693215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800706451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"":"downstairs","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800722758Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800735664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"0","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.80074951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"1","":"downstairs","session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800763557Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"client":"2","":"downstairs","session_id"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800781295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800794481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"d467e58b-6506-4bf3-ad93-75a08b01d25b is now active with session: 68d2db95-f216-4a3d-ab0f-9e1273180f04","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800809247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800821663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760,"session_id":"68d2db95-f216-4a3d-ab0f-9e1273180f04"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800834919Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} {"msg":"Activated sub_volume d467e58b-6506-4bf3-ad93-75a08b01d25b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:41.800846414Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 145 len 40960 data: 10 10 9 11 12 13 14 13 11 11 02/30 Write block 318 len 8192 data: 18 19 03/30 Read block 55 len 36864 04/30 Read block 313 len 24576 05/30 Flush 06/30 Write block 163 len 8192 data: 8 8 07/30 Read block 557 len 12288 08/30 Read block 129 len 12288 09/30 Read block 417 len 28672 10/30 Read block 112 len 32768 11/30 Read block 85 len 4096 12/30 Read block 315 len 28672 13/30 Write block 379 len 12288 data: 17 18 17 14/30 Read block 190 len 20480 15/30 Read block 573 len 28672 16/30 Read block 464 len 8192 17/30 Read block 457 len 4096 18/30 Read block 276 len 20480 19/30 Read block 565 len 40960 20/30 Write block 430 len 36864 data: 15 16 17 16 11 13 16 16 18 21/30 Read block 334 len 12288 22/30 Write block 309 len 32768 data: 10 10 11 10 9 9 12 15 23/30 Write block 409 len 16384 data: 14 13 14 15 24/30 Read block 174 len 20480 25/30 Read block 69 len 8192 26/30 Read block 539 len 24576 27/30 Read block 48 len 32768 28/30 Read block 313 len 24576 29/30 Read block 78 len 24576 30/30 Read block 106 len 24576 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Sent Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 3 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) None Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:41.813087336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1760} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 135 135 135 230 230 230 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 132 138 236 227 236 F F F <--- 8 160-179 137 137 137 235 235 235 T F T <--- 9 180-199 137 137 137 235 235 235 F F F 10 200-219 135 135 135 230 230 230 F F F 11 220-239 137 137 137 235 235 235 F F F 12 240-259 137 137 137 235 235 235 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 138 136 138 236 234 236 T F T <--- 16 320-339 136 136 136 234 234 234 F F F 17 340-359 136 136 136 232 232 232 F F F 18 360-379 135 135 135 230 230 230 T F T <--- 19 380-399 134 134 134 229 229 229 T F T <--- 20 400-419 132 132 132 227 227 227 T F T <--- 21 420-439 135 135 135 230 230 230 T F T <--- 22 440-459 136 136 136 234 234 234 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 134 134 134 229 229 229 F F F 28 560-579 136 136 136 234 234 234 F F F 29 580-599 136 136 136 234 234 234 F F F Max gen: 138, Max flush: 236 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 135 135 135 230 230 230 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 138 138 236 236 236 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 137 137 137 235 235 235 F F F 10 200-219 135 135 135 230 230 230 F F F 11 220-239 137 137 137 235 235 235 F F F 12 240-259 137 137 137 235 235 235 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 139 139 139 237 237 237 F F F 16 320-339 136 136 136 234 234 234 F F F 17 340-359 136 136 136 232 232 232 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 139 139 139 237 237 237 F F F 21 420-439 139 139 139 237 237 237 F F F 22 440-459 136 136 136 234 234 234 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 134 134 134 229 229 229 F F F 28 560-579 136 136 136 234 234 234 F F F 29 580-599 136 136 136 234 234 234 F F F Max gen: 139, Max flush: 237 Begin loop 70 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.129830227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.141434289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.141755355Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.141795583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.141807679Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.141825567Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"Crucible e98b8b89-298b-447a-9c10-4aaf8419b8fb has session id: b8ac63c8-dc02-4b8e-a810-07e60da49da1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142180238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Upstairs opts: Upstairs UUID: e98b8b89-298b-447a-9c10-4aaf8419b8fb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142208891Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Crucible stats registered with UUID: e98b8b89-298b-447a-9c10-4aaf8419b8fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142223888Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:46.142236924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142412404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14244754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142463307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142565674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142581231Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"e98b8b89-298b-447a-9c10-4aaf8419b8fb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142597228Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142610034Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14262402Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142638187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142651823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14266612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142680216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:41674)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142708569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:52995)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142723516Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:54539)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.142808074Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143482951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143531694Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143549121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143867967Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143912297Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[0]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 236, 237, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143950524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[0]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 138, 139, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.143967592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14398452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[1]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 236, 237, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144000427Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[1]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 138, 139, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144016464Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144032882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[2]R flush_numbers[0..12]: [230, 230, 232, 235, 234, 234, 230, 236, 237, 235, 230, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14405117Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[2]R generation[0..12]: [135, 135, 136, 137, 136, 136, 135, 138, 139, 137, 135, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144066947Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144082994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144103344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.14411777Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Next flush: 238","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144131146Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144144853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"":"downstairs","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144159279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144172115Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"0","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144186151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"clientDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"1","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144203199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"client":"2","":"downstairs","session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144217616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144230832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"e98b8b89-298b-447a-9c10-4aaf8419b8fb is now active with session: b8ac63c8-dc02-4b8e-a810-07e60da49da1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144245608Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144258514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768,"session_id":"b8ac63c8-dc02-4b8e-a810-07e60da49da1"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144273711Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} {"msg":"Activated sub_volume e98b8b89-298b-447a-9c10-4aaf8419b8fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:46.144285307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 549 len 16384 data: 12 15 13 11 02/30 Read block 297 len 28672 03/30 Read block 461 len 32768 04/30 Read block 47 len 8192 05/30 Flush 06/30 Write block 572 len 36864 data: 13 10 13 15 11 14 14 15 18 07/30 Write block 342 len 28672 data: 9 11 10 10 10 8 10 08/30 Read block 43 len 32768 09/30 Write block 26 len 12288 data: 8 6 6 10/30 Read block 455 len 16384 11/30 Read block 33 len 4096 12/30 Read block 219 len 28672 13/30 Read block 9 len 28672 14/30 Write block 345 len 16384 data: 11 11 9 11 15/30 Read block 386 len 24576 16/30 Write block 252 len 20480 data: 14 14 18 16 14 17/30 Write block 315 len 28672 data: 13 16 16 19 20 20 17 18/30 Read block 78 len 24576 19/30 Read block 479 len 8192 20/30 Flush 21/30 Write block 195 len 40960 data: 15 14 14 11 11 11 8 7 6 5 22/30 Write block 310 len 16384 data: 11 12 11 10 23/30 Write block 440 len 8192 data: 15 10 24/30 Read block 446 len 20480 25/30 Write block 188 len 4096 data: 14 26/30 Write block 345 len 40960 data: 12 12 10 12 13 11 13 12 11 12 27/30 Read block 316 len 36864 28/30 Read block 528 len 8192 29/30 Read block 217 len 4096 30/30 Write block 568 len 36864 data: 11 14 15 13 14 11 14 16 12 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Read 7 Done Sent Done false 1002 Acked Read 8 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:46.156447922Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1768} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 140 135 140 239 230 239 F F F <--- 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 138 138 236 236 236 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 137 137 137 235 235 235 T F T <--- 10 200-219 135 135 135 230 230 230 T F T <--- 11 220-239 137 137 137 235 235 235 F F F 12 240-259 140 137 140 239 235 239 F F F <--- 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 140 139 140 239 237 239 T F T <--- 16 320-339 140 136 140 239 234 239 F F F <--- 17 340-359 140 136 140 239 232 239 T F T <--- 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 139 139 139 237 237 237 F F F 21 420-439 139 139 139 237 237 237 F F F 22 440-459 136 136 136 234 234 234 T F T <--- 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 140 134 140 238 229 238 F F F <--- 28 560-579 140 136 140 239 234 239 T F T <--- 29 580-599 140 136 140 239 234 239 F F F <--- Max gen: 140, Max flush: 239 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 230 230 230 F F F 1 020-039 140 140 140 239 239 239 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 138 138 236 236 236 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F F F 11 220-239 137 137 137 235 235 235 F F F 12 240-259 140 140 140 239 239 239 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F F F 17 340-359 141 141 141 240 240 240 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 139 139 139 237 237 237 F F F 21 420-439 139 139 139 237 237 237 F F F 22 440-459 141 141 141 240 240 240 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 F F F 25 500-519 130 130 130 222 222 222 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 140 140 140 238 238 238 F F F 28 560-579 141 141 141 240 240 240 F F F 29 580-599 140 140 140 239 239 239 F F F Max gen: 141, Max flush: 240 Begin loop 71 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.479451774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491115342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491430145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491468043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491480128Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491498106Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"Crucible 5a1dbc47-31bb-4f60-bf31-9befefe21ce9 has session id: 001c3dd1-12e1-47e4-8d35-6bdeb787c822","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491868665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Upstairs opts: Upstairs UUID: 5a1dbc47-31bb-4f60-bf31-9befefe21ce9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491899139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Crucible stats registered with UUID: 5a1dbc47-31bb-4f60-bf31-9befefe21ce9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.491914256Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:50.491927602Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492105963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492128383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492146321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492273189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492288236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"5a1dbc47-31bb-4f60-bf31-9befefe21ce9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492310836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492323472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492337518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"1","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492351705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492365231Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492379427Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492393574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"ds_connection connected from Ok(127.0.0.1:59660)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.49245157Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"ds_connection connected from Ok(127.0.0.1:33836)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492468748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"ds_connection connected from Ok(127.0.0.1:54027)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.492534078Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493166045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493203683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"1","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493246152Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493536514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493596071Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[0]R flush_numbers[0..12]: [230, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.49365996Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[0]R generation[0..12]: [135, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493677938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493694916Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[1]R flush_numbers[0..12]: [230, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493710783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[1]R generation[0..12]: [135, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493726501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493743088Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[2]R flush_numbers[0..12]: [230, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 235]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493758845Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[2]R generation[0..12]: [135, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493774542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.49379072Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2026-03-14T05:05:50.493810449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493824735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Next flush: 241","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493837861Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493854199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"":"downstairs","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493868255Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493880921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"0","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493894507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"1","":"downstairs"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493911675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"client":"2","":"downstairs","session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493926211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493939648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"5a1dbc47-31bb-4f60-bf31-9befefe21ce9 is now active with session: 001c3dd1-12e1-47e4-8d35-6bdeb787c822","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493954694Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.49396735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776,"session_id":"001c3dd1-12e1-47e4-8d35-6bdeb787c822"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493980396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} {"msg":"Activated sub_volume 5a1dbc47-31bb-4f60-bf31-9befefe21ce9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:50.493992051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 410 len 32768 data: 14 15 16 13 9 10 8 8 02/30 Write block 251 len 20480 data: 12 15 15 19 17 03/30 Read block 308 len 24576 04/30 Write block 506 len 20480 data: 12 12 11 11 12 05/30 Flush 06/30 Write block 509 len 16384 data: 12 13 11 13 07/30 Write block 595 len 8192 data: 9 9 08/30 Flush 09/30 Write block 1 len 28672 data: 5 6 9 8 11 10 8 10/30 Read block 308 len 16384 11/30 Flush 12/30 Write block 570 len 24576 data: 16 14 15 12 15 17 13/30 Read block 207 len 20480 14/30 Read block 298 len 24576 15/30 Flush 16/30 Write block 562 len 40960 data: 7 8 10 10 12 10 12 15 17 15 17/30 Read block 81 len 16384 18/30 Write block 223 len 24576 data: 12 12 14 15 16 14 19/30 Write block 234 len 20480 data: 14 13 13 11 9 20/30 Flush 21/30 Write block 561 len 8192 data: 8 8 22/30 Read block 433 len 12288 23/30 Write block 423 len 12288 data: 13 12 15 24/30 Read block 352 len 4096 25/30 Read block 575 len 20480 26/30 Read block 322 len 36864 27/30 Read block 174 len 16384 28/30 Read block 236 len 32768 29/30 Write block 495 len 32768 data: 10 11 14 16 14 13 12 11 30/30 Read block 415 len 36864 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 5 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:50.506649262Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1776} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 135 142 243 230 243 F F F <--- 1 020-039 140 140 140 239 239 239 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 138 138 236 236 236 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F F F 11 220-239 142 137 142 245 235 245 F F F <--- 12 240-259 142 140 142 241 239 241 F F F <--- 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F F F 17 340-359 141 141 141 240 240 240 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 142 139 142 241 237 241 F T F <--- 21 420-439 139 139 139 237 237 237 T F T <--- 22 440-459 141 141 141 240 240 240 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 137 137 137 235 235 235 T F T <--- 25 500-519 142 130 142 242 222 242 T F T <--- 26 520-539 136 136 136 234 234 234 F F F 27 540-559 140 140 140 238 238 238 F F F 28 560-579 142 141 142 245 240 245 T F T <--- 29 580-599 142 140 142 242 239 242 F F F <--- Max gen: 142, Max flush: 245 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 243 243 243 F F F 1 020-039 140 140 140 239 239 239 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 136 136 136 234 234 234 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 138 138 138 236 236 236 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F F F 11 220-239 142 142 142 245 245 245 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F F F 17 340-359 141 141 141 240 240 240 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 142 142 142 241 241 241 F F F 21 420-439 143 143 143 246 246 246 F F F 22 440-459 141 141 141 240 240 240 F F F 23 460-479 136 136 136 232 232 232 F F F 24 480-499 143 143 143 246 246 246 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 140 140 140 238 238 238 F F F 28 560-579 143 143 143 246 246 246 F F F 29 580-599 142 142 142 242 242 242 F F F Max gen: 143, Max flush: 246 Begin loop 72 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.82303429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.834674278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.834993533Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835032671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835044896Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835066036Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"Crucible 3d88d602-e92e-4fd3-ba83-4cf3dfaa8411 has session id: 0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835447209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Upstairs opts: Upstairs UUID: 3d88d602-e92e-4fd3-ba83-4cf3dfaa8411, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835482385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Crucible stats registered with UUID: 3d88d602-e92e-4fd3-ba83-4cf3dfaa8411","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835497823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:54.835511189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835674583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835705387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835720534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835834576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835850273Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"3d88d602-e92e-4fd3-ba83-4cf3dfaa8411 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835880647Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835894623Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835908649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835922716Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"2","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835936692Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.83595368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835967997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:58805)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.835982713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:64318)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.836004383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:64777)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.83601931Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.83678857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.836817683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"2","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.836841894Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837180538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.83722749Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[0]R flush_numbers[0..12]: [243, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837291329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[0]R generation[0..12]: [142, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837308517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837325014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[1]R flush_numbers[0..12]: [243, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837340721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[1]R generation[0..12]: [142, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837356729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837375827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[2]R flush_numbers[0..12]: [243, 239, 232, 235, 234, 234, 230, 236, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837391684Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[2]R generation[0..12]: [142, 140, 136, 137, 136, 136, 135, 138, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837407522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837423229Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837436885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837457124Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Next flush: 247","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.83747057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837484057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"":"downstairs","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837498463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837511279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"0","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837525735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"1","":"downstairs","session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837539932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837556139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837569315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"3d88d602-e92e-4fd3-ba83-4cf3dfaa8411 is now active with session: 0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837584252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837598609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784,"session_id":"0bb72d37-8ee3-4e94-af0e-ca7b0d3cf5bc"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837619608Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} {"msg":"Activated sub_volume 3d88d602-e92e-4fd3-ba83-4cf3dfaa8411","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:54.837631063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 579 len 36864 data: 16 19 18 16 15 16 14 11 7 02/30 Flush 03/30 Write block 454 len 40960 data: 9 7 6 8 9 8 8 9 8 10 04/30 Read block 397 len 32768 05/30 Write block 87 len 20480 data: 11 10 13 16 18 06/30 Read block 148 len 8192 07/30 Read block 349 len 8192 08/30 Read block 353 len 32768 09/30 Flush 10/30 Write block 271 len 24576 data: 12 13 13 12 12 14 11/30 Write block 152 len 4096 data: 14 12/30 Read block 104 len 12288 13/30 Write block 444 len 4096 data: 9 14/30 Read block 454 len 24576 15/30 Read block 311 len 40960 16/30 Read block 190 len 36864 17/30 Flush 18/30 Flush 19/30 Write block 270 len 12288 data: 11 13 14 20/30 Read block 137 len 20480 21/30 Flush 22/30 Flush 23/30 Write block 404 len 20480 data: 5 5 7 11 11 24/30 Read block 446 len 28672 25/30 Write block 550 len 4096 data: 16 26/30 Read block 573 len 28672 27/30 Write block 561 len 8192 data: 9 9 28/30 Read block 379 len 20480 29/30 Write block 140 len 16384 data: 12 9 10 11 30/30 Read block 255 len 4096 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 10 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 2 26 Done 28 4 26 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1001)) Some(JobId(1021)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:54.849765596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1784} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 243 243 243 F F F 1 020-039 140 140 140 239 239 239 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 144 136 144 248 234 248 F T F <--- 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 144 138 144 249 236 249 T F T <--- 8 160-179 139 139 139 237 237 237 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F F F 11 220-239 142 142 142 245 245 245 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 144 134 144 251 229 251 F F F <--- 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F F F 17 340-359 141 141 141 240 240 240 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 142 142 142 241 241 241 T F T <--- 21 420-439 143 143 143 246 246 246 F F F 22 440-459 144 141 144 249 240 249 F T F <--- 23 460-479 144 136 144 248 232 248 F T F <--- 24 480-499 143 143 143 246 246 246 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 140 140 140 238 238 238 T F T <--- 28 560-579 144 144 144 247 247 247 T F T <--- 29 580-599 144 144 144 247 247 247 F F F Max gen: 144, Max flush: 251 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 243 243 243 F F F 1 020-039 140 140 140 239 239 239 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F F F 4 080-099 144 144 144 248 248 248 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 145 145 145 252 252 252 F F F 8 160-179 139 139 139 237 237 237 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F F F 11 220-239 142 142 142 245 245 245 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 144 144 144 251 251 251 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F F F 17 340-359 141 141 141 240 240 240 F F F 18 360-379 139 139 139 237 237 237 F F F 19 380-399 139 139 139 237 237 237 F F F 20 400-419 145 145 145 252 252 252 F F F 21 420-439 143 143 143 246 246 246 F F F 22 440-459 144 144 144 249 249 249 F F F 23 460-479 144 144 144 248 248 248 F F F 24 480-499 143 143 143 246 246 246 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 136 136 136 234 234 234 F F F 27 540-559 145 145 145 252 252 252 F F F 28 560-579 145 145 145 252 252 252 F F F 29 580-599 144 144 144 247 247 247 F F F Max gen: 145, Max flush: 252 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.167580836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:59.178229342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.178276324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.188708932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189003736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189041693Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189054079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189072037Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"Crucible de3eaf49-661a-40f8-bcf9-90c59f7e3acd has session id: 7ce7aded-be9b-4c71-8096-4319c37fc3a4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189440855Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Upstairs opts: Upstairs UUID: de3eaf49-661a-40f8-bcf9-90c59f7e3acd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189475221Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Crucible stats registered with UUID: de3eaf49-661a-40f8-bcf9-90c59f7e3acd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189491358Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:59.189504764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189667418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189691329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189706146Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189830943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.18984762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"de3eaf49-661a-40f8-bcf9-90c59f7e3acd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189865028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189877784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.18989155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189905457Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"2","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189953328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.18997875Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.189994217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:53099)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190009474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:50097)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190024931Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:63680)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190138102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190773252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"2","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190804266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.190821764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.19114236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.19118627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[0]R flush_numbers[0..12]: [243, 239, 232, 235, 248, 234, 230, 252, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191245547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[0]R generation[0..12]: [142, 140, 136, 137, 144, 136, 135, 145, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191263495Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191280222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[1]R flush_numbers[0..12]: [243, 239, 232, 235, 248, 234, 230, 252, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.19129644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[1]R generation[0..12]: [142, 140, 136, 137, 144, 136, 135, 145, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191312397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191329095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[2]R flush_numbers[0..12]: [243, 239, 232, 235, 248, 234, 230, 252, 237, 240, 240, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191344842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[2]R generation[0..12]: [142, 140, 136, 137, 144, 136, 135, 145, 139, 141, 141, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191360649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191376776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:59.191390172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191412493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Next flush: 253","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191426129Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191439725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"":"downstairs","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191454041Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191466937Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"0","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191481324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"1","":"downstairs","session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.19149541Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191511778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191524954Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"de3eaf49-661a-40f8-bcf9-90c59f7e3acd is now active with session: 7ce7aded-be9b-4c71-8096-4319c37fc3a4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.19153977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191558959Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792,"session_id":"7ce7aded-be9b-4c71-8096-4319c37fc3a4"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191572565Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} {"msg":"Activated sub_volume de3eaf49-661a-40f8-bcf9-90c59f7e3acd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:59.191584081Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 163 len 20480 data: 9 9 9 11 13 02/30 Read block 7 len 16384 03/30 Read block 188 len 32768 04/30 Flush 05/30 Write block 564 len 36864 data: 11 11 13 11 13 16 18 16 16 06/30 Write block 358 len 28672 data: 15 13 12 8 9 9 11 07/30 Write block 232 len 12288 data: 16 15 15 08/30 Read block 151 len 32768 09/30 Write block 549 len 36864 data: 13 17 14 12 14 12 11 10 8 10/30 Write block 353 len 36864 data: 12 13 13 13 13 16 14 13 9 11/30 Read block 362 len 12288 12/30 Write block 62 len 28672 data: 9 9 11 10 12 12 13 13/30 Write block 31 len 28672 data: 9 8 9 12 10 9 8 14/30 Write block 575 len 36864 data: 18 13 15 15 17 20 19 17 16 15/30 Read block 68 len 4096 16/30 Read block 343 len 36864 17/30 Write block 582 len 8192 data: 18 17 18/30 Write block 336 len 40960 data: 10 13 13 13 14 14 10 12 11 13 19/30 Write block 534 len 28672 data: 10 12 9 8 11 15 14 20/30 Read block 459 len 28672 21/30 Read block 262 len 8192 22/30 Read block 59 len 8192 23/30 Read block 179 len 40960 24/30 Read block 397 len 8192 25/30 Read block 375 len 12288 26/30 Write block 205 len 8192 data: 5 6 27/30 Write block 320 len 36864 data: 21 18 16 15 14 13 15 12 11 28/30 Write block 428 len 28672 data: 13 13 16 17 18 17 12 29/30 Read block 573 len 28672 30/30 Write block 389 len 32768 data: 14 13 13 14 15 15 11 13 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:59.203944856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1792} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 243 243 243 F F F 1 020-039 140 140 140 239 239 239 F T T <--- 2 040-059 136 136 136 232 232 232 F F F 3 060-079 137 137 137 235 235 235 F T T <--- 4 080-099 144 144 144 248 248 248 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 145 145 145 252 252 252 F F F 8 160-179 139 146 146 237 253 253 F F F <--- 9 180-199 141 141 141 240 240 240 F F F 10 200-219 141 141 141 240 240 240 F T T <--- 11 220-239 142 142 142 245 245 245 F T T <--- 12 240-259 142 142 142 241 241 241 F F F 13 260-279 144 144 144 251 251 251 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 140 140 140 239 239 239 F T T <--- 17 340-359 141 141 141 240 240 240 F T T <--- 18 360-379 139 139 139 237 237 237 F T T <--- 19 380-399 139 139 139 237 237 237 F T T <--- 20 400-419 145 145 145 252 252 252 F F F 21 420-439 143 143 143 246 246 246 F T T <--- 22 440-459 144 144 144 249 249 249 F F F 23 460-479 144 144 144 248 248 248 F F F 24 480-499 143 143 143 246 246 246 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 136 136 136 234 234 234 F T T <--- 27 540-559 145 145 145 252 252 252 F T T <--- 28 560-579 145 145 145 252 252 252 F T T <--- 29 580-599 144 144 144 247 247 247 F T T <--- Max gen: 146, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 243 243 243 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 147 147 147 254 254 254 F F F 4 080-099 144 144 144 248 248 248 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 145 145 145 252 252 252 F F F 8 160-179 146 146 146 253 253 253 F F F 9 180-199 141 141 141 240 240 240 F F F 10 200-219 147 147 147 254 254 254 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 144 144 144 251 251 251 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 147 147 147 254 254 254 F F F 18 360-379 147 147 147 254 254 254 F F F 19 380-399 147 147 147 254 254 254 F F F 20 400-419 145 145 145 252 252 252 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 144 144 144 249 249 249 F F F 23 460-479 144 144 144 248 248 248 F F F 24 480-499 143 143 143 246 246 246 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 147 147 147 254 254 254 F F F 27 540-559 147 147 147 254 254 254 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 147, Max flush: 254 Begin loop 74 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.547462535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.559486697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.559838527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.559880016Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.559892682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.55991061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"Crucible 77ed8020-7e86-4bc9-9325-fea01e9bba60 has session id: f8881c0e-faaf-4cb3-a93f-1d0273a1e941","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560256588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Upstairs opts: Upstairs UUID: 77ed8020-7e86-4bc9-9325-fea01e9bba60, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560291103Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Crucible stats registered with UUID: 77ed8020-7e86-4bc9-9325-fea01e9bba60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560307321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:03.560320747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560498368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560533534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560549151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560670096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560685053Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"77ed8020-7e86-4bc9-9325-fea01e9bba60 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560704271Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560716797Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560733675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560747911Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560776294Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.56081102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.5608336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"ds_connection connected from Ok(127.0.0.1:57699)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560849337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"ds_connection connected from Ok(127.0.0.1:61652)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560864284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"ds_connection connected from Ok(127.0.0.1:39107)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.560955015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.561699424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.561736351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.561753389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562087951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562134963Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[0]R flush_numbers[0..12]: [243, 254, 232, 254, 248, 234, 230, 252, 253, 240, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562194049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[0]R generation[0..12]: [142, 147, 136, 147, 144, 136, 135, 145, 146, 141, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562211587Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562231637Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[1]R flush_numbers[0..12]: [243, 254, 232, 254, 248, 234, 230, 252, 253, 240, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562247634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[1]R generation[0..12]: [142, 147, 136, 147, 144, 136, 135, 145, 146, 141, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562262981Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562279288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[2]R flush_numbers[0..12]: [243, 254, 232, 254, 248, 234, 230, 252, 253, 240, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562294825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[2]R generation[0..12]: [142, 147, 136, 147, 144, 136, 135, 145, 146, 141, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562310312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.56232693Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:06:03.562340466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562359315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562372681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562385987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"":"downstairs","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562400293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562412799Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"0","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562426995Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"1","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562441102Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"client":"2","":"downstairs","session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.56245962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562473647Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"77ed8020-7e86-4bc9-9325-fea01e9bba60 is now active with session: f8881c0e-faaf-4cb3-a93f-1d0273a1e941","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562488854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562501829Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800,"session_id":"f8881c0e-faaf-4cb3-a93f-1d0273a1e941"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562515396Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} {"msg":"Activated sub_volume 77ed8020-7e86-4bc9-9325-fea01e9bba60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:03.562526881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 456 len 32768 data: 7 9 10 9 9 10 9 11 03/30 Read block 181 len 16384 04/30 Read block 80 len 36864 05/30 Write block 1 len 24576 data: 6 7 10 9 12 11 06/30 Read block 564 len 24576 07/30 Read block 440 len 36864 08/30 Write block 145 len 16384 data: 11 11 10 12 09/30 Read block 395 len 28672 10/30 Read block 587 len 20480 11/30 Read block 328 len 8192 12/30 Read block 285 len 12288 13/30 Read block 169 len 28672 14/30 Write block 377 len 36864 data: 13 15 18 19 18 20 22 19 16 15/30 Read block 477 len 4096 16/30 Write block 162 len 20480 data: 5 10 10 10 12 17/30 Flush 18/30 Write block 182 len 20480 data: 10 10 11 13 14 19/30 Read block 343 len 28672 20/30 Flush 21/30 Write block 140 len 40960 data: 13 10 11 12 12 12 12 11 13 13 22/30 Write block 467 len 40960 data: 9 8 9 9 11 10 11 9 12 9 23/30 Write block 270 len 4096 data: 12 24/30 Write block 476 len 12288 data: 10 13 14 25/30 Read block 55 len 8192 26/30 Read block 171 len 4096 27/30 Write block 488 len 20480 data: 10 14 14 13 11 28/30 Write block 166 len 32768 data: 13 14 13 14 15 16 17 16 29/30 Read block 323 len 32768 30/30 Read block 521 len 20480 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Read 2 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 29 28 0 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:03.574913298Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1800} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 142 256 256 243 F F F <--- 1 020-039 147 147 147 254 254 254 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 147 147 147 254 254 254 F F F 4 080-099 144 144 144 248 248 248 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 148 148 145 256 256 252 T T F <--- 8 160-179 148 148 146 256 256 253 T T F <--- 9 180-199 148 148 141 257 257 240 F F F <--- 10 200-219 147 147 147 254 254 254 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 144 144 144 251 251 251 T T F <--- 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 147 147 147 254 254 254 F F F 18 360-379 148 148 147 256 256 254 F F F <--- 19 380-399 148 148 147 256 256 254 F F F <--- 20 400-419 145 145 145 252 252 252 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 144 256 256 249 F F F <--- 23 460-479 148 148 144 256 256 248 T T F <--- 24 480-499 143 143 143 246 246 246 T T F <--- 25 500-519 143 143 143 246 246 246 F F F 26 520-539 147 147 147 254 254 254 F F F 27 540-559 147 147 147 254 254 254 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 148, Max flush: 257 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 256 256 256 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 136 136 136 232 232 232 F F F 3 060-079 147 147 147 254 254 254 F F F 4 080-099 144 144 144 248 248 248 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 149 149 149 258 258 258 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 148 148 257 257 257 F F F 10 200-219 147 147 147 254 254 254 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 142 142 142 241 241 241 F F F 13 260-279 149 149 149 258 258 258 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 147 147 147 254 254 254 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 145 145 145 252 252 252 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 148 256 256 256 F F F 23 460-479 149 149 149 258 258 258 F F F 24 480-499 149 149 149 258 258 258 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 147 147 147 254 254 254 F F F 27 540-559 147 147 147 254 254 254 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 149, Max flush: 258 Begin loop 75 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.899264594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.910864363Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911182848Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911220015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911232701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911250339Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"Crucible b84ca768-5d7a-4c25-9064-afa2d5934905 has session id: 62bd6b73-9b8e-45a0-874b-e32cf946f3a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911630422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Upstairs opts: Upstairs UUID: b84ca768-5d7a-4c25-9064-afa2d5934905, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911661206Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Crucible stats registered with UUID: b84ca768-5d7a-4c25-9064-afa2d5934905","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911679174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:07.911698593Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911874803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911909459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.911925206Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912019319Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912034726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"b84ca768-5d7a-4c25-9064-afa2d5934905 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912055045Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912067971Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912082167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912095944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.91210938Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912123746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912138103Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:56179)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912152649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:57335)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912167446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:44893)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912182493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912907283Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912947291Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.912993622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913271909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913310446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[0]R flush_numbers[0..12]: [256, 254, 232, 254, 248, 234, 230, 258, 258, 257, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913372154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[0]R generation[0..12]: [148, 147, 136, 147, 144, 136, 135, 149, 149, 148, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913389452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.91340622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[1]R flush_numbers[0..12]: [256, 254, 232, 254, 248, 234, 230, 258, 258, 257, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913422127Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[1]R generation[0..12]: [148, 147, 136, 147, 144, 136, 135, 149, 149, 148, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913437924Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913454452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[2]R flush_numbers[0..12]: [256, 254, 232, 254, 248, 234, 230, 258, 258, 257, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913470219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[2]R generation[0..12]: [148, 147, 136, 147, 144, 136, 135, 149, 149, 148, 147, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913485416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913504865Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913518841Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs",""Wait for a query_work_queue command to finish before sending IO :"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913539611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Next flush: 259","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913552847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913565713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"":"downstairs","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913580979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913593555Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"0","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913607582Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"1","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913621738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"client":"2","":"downstairs","session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:07.913636565Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913651862Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"b84ca768-5d7a-4c25-9064-afa2d5934905 is now active with session: 62bd6b73-9b8e-45a0-874b-e32cf946f3a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913666668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913679754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808,"session_id":"62bd6b73-9b8e-45a0-874b-e32cf946f3a9"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913693181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} {"msg":"Activated sub_volume b84ca768-5d7a-4c25-9064-afa2d5934905","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:07.913704676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 206 len 28672 data: 7 6 6 5 8 9 10 02/30 Read block 173 len 24576 03/30 Write block 435 len 16384 data: 14 17 17 19 04/30 Write block 419 len 28672 data: 7 9 9 12 14 13 16 05/30 Read block 235 len 12288 06/30 Write block 445 len 16384 data: 11 9 7 11 07/30 Write block 8 len 16384 data: 12 12 11 9 08/30 Read block 357 len 20480 09/30 Write block 46 len 24576 data: 11 12 14 12 12 15 10/30 Flush 11/30 Write block 40 len 4096 data: 8 12/30 Write block 525 len 4096 data: 8 13/30 Flush 14/30 Write block 85 len 24576 data: 11 13 12 11 14 17 15/30 Read block 553 len 28672 16/30 Read block 335 len 40960 17/30 Read block 258 len 8192 18/30 Flush 19/30 Flush 20/30 Write block 10 len 8192 data: 12 10 21/30 Read block 103 len 12288 22/30 Read block 201 len 36864 23/30 Write block 249 len 20480 data: 17 15 13 16 16 24/30 Write block 97 len 4096 data: 11 25/30 Read block 372 len 4096 26/30 Read block 275 len 36864 27/30 Write block 552 len 28672 data: 13 15 13 12 11 9 8 28/30 Read block 28 len 24576 29/30 Read block 90 len 8192 30/30 Write block 205 len 12288 data: 6 8 7 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:07.925303075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1808} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 148 150 259 256 259 T F T <--- 1 020-039 147 147 147 254 254 254 F F F 2 040-059 150 136 150 260 232 260 F F F <--- 3 060-079 147 147 147 254 254 254 F F F 4 080-099 150 144 150 261 248 261 T F T <--- 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 149 149 149 258 258 258 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 148 148 257 257 257 F F F 10 200-219 150 147 150 259 254 259 T T T <--- 11 220-239 147 147 147 254 254 254 F F F 12 240-259 142 142 142 241 241 241 T F T <--- 13 260-279 149 149 149 258 258 258 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 147 147 147 254 254 254 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 145 150 259 252 259 F F F <--- 21 420-439 150 147 150 259 254 259 F T F <--- 22 440-459 150 148 150 259 256 259 F F F <--- 23 460-479 149 149 149 258 258 258 F F F 24 480-499 149 149 149 258 258 258 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 150 147 150 260 254 260 F F F <--- 27 540-559 147 147 147 254 254 254 T F T <--- 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 150, Max flush: 261 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 262 262 262 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 150 150 150 260 260 260 F F F 3 060-079 147 147 147 254 254 254 F F F 4 080-099 151 151 151 262 262 262 F F F 5 100-119 136 136 136 234 234 234 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 149 149 149 258 258 258 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 148 148 257 257 257 F F F 10 200-219 151 151 151 262 262 262 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 151 151 151 262 262 262 F F F 13 260-279 149 149 149 258 258 258 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 147 147 147 254 254 254 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 150 150 150 259 259 259 F F F 22 440-459 150 150 150 259 259 259 F F F 23 460-479 149 149 149 258 258 258 F F F 24 480-499 149 149 149 258 258 258 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 150 150 150 260 260 260 F F F 27 540-559 151 151 151 262 262 262 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 151, Max flush: 262 Begin loop 76 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.245865215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.257507945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.25782729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.257869329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.257882085Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.257900443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"Crucible a23f554f-6a9c-47d7-a64e-e05fd7473aa0 has session id: 5a5d40d1-44f4-4efc-bc3e-eaa3a854b592","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258281597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Upstairs opts: Upstairs UUID: a23f554f-6a9c-47d7-a64e-e05fd7473aa0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258315352Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Crucible stats registered with UUID: a23f554f-6a9c-47d7-a64e-e05fd7473aa0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258331189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:12.258344205Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258495334Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.25852974Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258545377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"2","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258658648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258673965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"a23f554f-6a9c-47d7-a64e-e05fd7473aa0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258691753Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.25870746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258721927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258735773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"2","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258763696Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"2","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258778783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258792789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"ds_connection connected from Ok(127.0.0.1:41539)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258807736Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"ds_connection connected from Ok(127.0.0.1:61396)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258828676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"ds_connection connected from Ok(127.0.0.1:45772)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.258855708Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"io task","client":"2","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.259539999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"2","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.259559778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.259575956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.259907707Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.259951497Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[0]R flush_numbers[0..12]: [262, 254, 260, 254, 262, 234, 230, 258, 258, 257, 262, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.26002415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[0]R generation[0..12]: [151, 147, 150, 147, 151, 136, 135, 149, 149, 148, 151, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260044099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260060826Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[1]R flush_numbers[0..12]: [262, 254, 260, 254, 262, 234, 230, 258, 258, 257, 262, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260076754Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[1]R generation[0..12]: [151, 147, 150, 147, 151, 136, 135, 149, 149, 148, 151, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260092811Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260109118Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[2]R flush_numbers[0..12]: [262, 254, 260, 254, 262, 234, 230, 258, 258, 257, 262, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260124796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[2]R generation[0..12]: [151, 147, 150, 147, 151, 136, 135, 149, 149, 148, 151, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260140143Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.26015617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:06:12.260169456Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260189435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Next flush: 263","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260203021Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260216977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"":"downstairs","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260231014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.26024393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"0","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260257956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"1","":"downstairs","session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260274023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260291141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260304337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"a23f554f-6a9c-47d7-a64e-e05fd7473aa0 is now active with session: 5a5d40d1-44f4-4efc-bc3e-eaa3a854b592","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260319104Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.26033198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816,"session_id":"5a5d40d1-44f4-4efc-bc3e-eaa3a854b592"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260345586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} {"msg":"Activated sub_volume a23f554f-6a9c-47d7-a64e-e05fd7473aa0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:12.260357101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 545 len 4096 data: 11 02/30 Write block 461 len 40960 data: 11 10 12 13 11 10 10 9 10 10 03/30 Read block 495 len 20480 04/30 Read block 273 len 12288 05/30 Flush 06/30 Write block 5 len 16384 data: 13 12 9 13 07/30 Read block 67 len 12288 08/30 Read block 257 len 8192 09/30 Write block 112 len 28672 data: 11 10 8 9 10 10 10 10/30 Write block 275 len 16384 data: 13 15 14 13 11/30 Write block 252 len 20480 data: 17 17 20 18 15 12/30 Write block 432 len 24576 data: 19 18 13 15 18 18 13/30 Read block 2 len 16384 14/30 Flush 15/30 Write block 210 len 24576 data: 9 10 11 9 10 11 16/30 Write block 349 len 36864 data: 14 12 14 13 13 14 14 14 14 17/30 Read block 446 len 4096 18/30 Write block 240 len 28672 data: 5 5 5 7 10 12 15 19/30 Write block 273 len 8192 data: 14 13 20/30 Flush 21/30 Write block 41 len 20480 data: 6 8 7 9 9 22/30 Read block 5 len 16384 23/30 Read block 50 len 32768 24/30 Read block 543 len 8192 25/30 Read block 38 len 32768 26/30 Read block 220 len 40960 27/30 Write block 202 len 32768 data: 8 7 6 7 9 8 7 6 28/30 Read block 143 len 24576 29/30 Read block 569 len 36864 30/30 Write block 446 len 4096 data: 10 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 4 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Read 2 Done Done Done false 1008 Acked Write 7 Done Done Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 21 1 23 Done 24 4 24 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1004)) Some(JobId(1019)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:12.27215337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1816} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 151 152 264 262 264 F T F <--- 1 020-039 147 147 147 254 254 254 F F F 2 040-059 150 150 150 260 260 260 T F T <--- 3 060-079 147 147 147 254 254 254 F F F 4 080-099 151 151 151 262 262 262 F F F 5 100-119 152 136 152 264 234 264 F T F <--- 6 120-139 135 135 135 230 230 230 F F F 7 140-159 149 149 149 258 258 258 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 148 148 257 257 257 F F F 10 200-219 152 151 152 265 262 265 T F T <--- 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 151 152 265 262 265 F F F <--- 13 260-279 152 149 152 265 258 265 F F F <--- 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 152 147 152 265 254 265 F F F <--- 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 150 152 264 259 264 F F F <--- 22 440-459 150 150 150 259 259 259 T F T <--- 23 460-479 152 152 152 263 263 263 F F F 24 480-499 149 149 149 258 258 258 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 150 150 150 260 260 260 F F F 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 152, Max flush: 265 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 264 264 264 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 147 147 147 254 254 254 F F F 4 080-099 151 151 151 262 262 262 F F F 5 100-119 152 152 152 264 264 264 F F F 6 120-139 135 135 135 230 230 230 F F F 7 140-159 149 149 149 258 258 258 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 148 148 257 257 257 F F F 10 200-219 153 153 153 266 266 266 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 152 152 265 265 265 F F F 13 260-279 152 152 152 265 265 265 F F F 14 280-299 137 137 137 235 235 235 F F F 15 300-319 141 141 141 240 240 240 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 152 152 152 265 265 265 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 152 152 152 263 263 263 F F F 24 480-499 149 149 149 258 258 258 F F F 25 500-519 143 143 143 246 246 246 F F F 26 520-539 150 150 150 260 260 260 F F F 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 153, Max flush: 266 Begin loop 77 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.59159422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:16.602211232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.602254581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.612627042Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.612909151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.612946458Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.612958583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.612976331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"Crucible aff119aa-9c3e-4fd6-a19a-91121b39dd11 has session id: 7f137327-1be8-44d1-8966-88ea2024dbc6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.61330363Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Upstairs opts: Upstairs UUID: aff119aa-9c3e-4fd6-a19a-91121b39dd11, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613340867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Crucible stats registered with UUID: aff119aa-9c3e-4fd6-a19a-91121b39dd11","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613356434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:16.61336977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613532274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.61356668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613582537Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613711756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613727043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"aff119aa-9c3e-4fd6-a19a-91121b39dd11 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613744451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613757427Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613771213Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613785149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613825748Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613840795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613854771Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"ds_connection connected from Ok(127.0.0.1:54243)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.613879122Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"ds_connection connected from Ok(127.0.0.1:40359)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.61389721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"ds_connection connected from Ok(127.0.0.1:59675)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.614058014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"io task","client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.614639268Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.614678576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.614804383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615020402Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615043832Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[0]R flush_numbers[0..12]: [264, 254, 266, 254, 262, 264, 230, 258, 258, 257, 266, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615181305Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[0]R generation[0..12]: [152, 147, 153, 147, 151, 152, 135, 149, 149, 148, 153, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615198853Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615215581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[1]R flush_numbers[0..12]: [264, 254, 266, 254, 262, 264, 230, 258, 258, 257, 266, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615231288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[1]R generation[0..12]: [152, 147, 153, 147, 151, 152, 135, 149, 149, 148, 153, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615247065Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615263502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[2]R flush_numbers[0..12]: [264, 254, 266, 254, 262, 264, 230, 258, 258, 257, 266, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615279149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[2]R generation[0..12]: [152, 147, 153, 147, 151, 152, 135, 149, 149, 148, 153, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615298028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615314285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":Wait for a query_work_queue command to finish before sending IO "Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615333344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615354294Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Next flush: 267","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.61536806Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615381776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"":"downstairs","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615395803Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615408408Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"0","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615422485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"1","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msgDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615439232Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"client":"2","":"downstairs","session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615453569Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615467005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"aff119aa-9c3e-4fd6-a19a-91121b39dd11 is now active with session: 7f137327-1be8-44d1-8966-88ea2024dbc6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615481692Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615494067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824,"session_id":"7f137327-1be8-44d1-8966-88ea2024dbc6"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615507333Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} {"msg":"Activated sub_volume aff119aa-9c3e-4fd6-a19a-91121b39dd11","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:16.615520429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 145 len 32768 data: 13 13 12 14 14 14 15 15 02/30 Read block 385 len 32768 03/30 Read block 95 len 40960 04/30 Read block 145 len 8192 05/30 Write block 69 len 4096 data: 14 06/30 Write block 192 len 8192 data: 13 10 07/30 Write block 492 len 12288 data: 12 12 14 08/30 Read block 589 len 36864 09/30 Write block 130 len 16384 data: 16 21 17 18 10/30 Write block 490 len 28672 data: 15 14 13 13 15 11 12 11/30 Read block 305 len 16384 12/30 Flush 13/30 Write block 13 len 4096 data: 11 14/30 Write block 103 len 16384 data: 7 7 5 7 15/30 Write block 142 len 32768 data: 12 13 13 14 14 13 15 15 16/30 Read block 571 len 4096 17/30 Read block 357 len 24576 18/30 Read block 460 len 20480 19/30 Read block 553 len 4096 20/30 Read block 586 len 36864 21/30 Read block 254 len 28672 22/30 Flush 23/30 Write block 493 len 16384 data: 14 16 12 13 24/30 Write block 279 len 8192 data: 13 11 25/30 Write block 290 len 12288 data: 16 15 14 26/30 Write block 108 len 36864 data: 13 10 12 12 12 11 9 10 11 27/30 Write block 304 len 20480 data: 5 7 8 11 12 28/30 Write block 209 len 36864 data: 7 10 11 12 10 11 12 13 13 29/30 Read block 400 len 12288 30/30 Write block 516 len 40960 data: 10 10 7 6 6 5 6 7 7 9 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:16.626658979Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1824} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 154 154 264 268 268 F F F <--- 1 020-039 147 147 147 254 254 254 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 147 154 154 254 267 267 F F F <--- 4 080-099 151 151 151 262 262 262 F F F 5 100-119 152 154 154 264 268 268 F T T <--- 6 120-139 135 154 154 230 267 267 F F F <--- 7 140-159 149 154 154 258 268 268 F F F <--- 8 160-179 149 149 149 258 258 258 F F F 9 180-199 148 154 154 257 267 267 F F F <--- 10 200-219 153 153 153 266 266 266 F T T <--- 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 152 152 265 265 265 F F F 13 260-279 152 152 152 265 265 265 F T T <--- 14 280-299 137 137 137 235 235 235 F T T <--- 15 300-319 141 141 141 240 240 240 F T T <--- 16 320-339 147 147 147 254 254 254 F F F 17 340-359 152 152 152 265 265 265 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 152 152 152 263 263 263 F F F 24 480-499 149 154 154 258 267 267 F T T <--- 25 500-519 143 143 143 246 246 246 F T T <--- 26 520-539 150 150 150 260 260 260 F T T <--- 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 154, Max flush: 268 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 268 268 268 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 154 154 154 267 267 267 F F F 4 080-099 151 151 151 262 262 262 F F F 5 100-119 155 155 155 269 269 269 F F F 6 120-139 154 154 154 267 267 267 F F F 7 140-159 154 154 154 268 268 268 F F F 8 160-179 149 149 149 258 258 258 F F F 9 180-199 154 154 154 267 267 267 F F F 10 200-219 155 155 155 269 269 269 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 152 152 265 265 265 F F F 13 260-279 155 155 155 269 269 269 F F F 14 280-299 155 155 155 269 269 269 F F F 15 300-319 155 155 155 269 269 269 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 152 152 152 265 265 265 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 152 152 152 263 263 263 F F F 24 480-499 155 155 155 269 269 269 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 155 155 155 269 269 269 F F F 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 155, Max flush: 269 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:20.996539496Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.008593342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.008924513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.00896309Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.008975556Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009007661Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"Crucible 4dbd65ca-97ab-4433-af11-d4eebbebd5f4 has session id: bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009393936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Upstairs opts: Upstairs UUID: 4dbd65ca-97ab-4433-af11-d4eebbebd5f4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.00942466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Crucible stats registered with UUID: 4dbd65ca-97ab-4433-af11-d4eebbebd5f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009440047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:21.009453303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009626752Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009652884Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009667871Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009769137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009784414Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"4dbd65ca-97ab-4433-af11-d4eebbebd5f4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009807154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.00982026Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009834247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009848153Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009862079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009876476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.009913583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"ds_connection connected from Ok(127.0.0.1:60059)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.00992982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"ds_connection connected from Ok(127.0.0.1:38483)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.010054517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"ds_connection connected from Ok(127.0.0.1:62376)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.010100948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"io task","client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.010659412Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.010695149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.01074209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.0110723Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011117051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[0]R flush_numbers[0..12]: [268, 254, 266, 267, 262, 269, 267, 268, 258, 267, 269, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011163692Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[0]R generation[0..12]: [154, 147, 153, 154, 151, 155, 154, 154, 149, 154, 155, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.01118095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011197887Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[1]R flush_numbers[0..12]: [268, 254, 266, 267, 262, 269, 267, 268, 258, 267, 269, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011213725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[1]R generation[0..12]: [154, 147, 153, 154, 151, 155, 154, 154, 149, 154, 155, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011229572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011245799Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[2]R flush_numbers[0..12]: [268, 254, 266, 267, 262, 269, 267, 268, 258, 267, 269, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011261937Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[2]R generation[0..12]: [154, 147, 153, 154, 151, 155, 154, 154, 149, 154, 155, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011278064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011294702Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011308828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Generation requested: 156 >= found:156","Wait for a query_work_queue command to finish before sending IO v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011329988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Next flush: 270","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011344144Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.01135766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"":"downstairs","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011374198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011388894Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"0","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011402691Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"1","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011416597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"client":"2","":"downstairs","session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:21.011431044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.01144583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"4dbd65ca-97ab-4433-af11-d4eebbebd5f4 is now active with session: bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011460667Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011473463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832,"session_id":"bdcf9f17-8ecc-4159-b1b1-ec2674fd0a1e"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011487389Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} {"msg":"Activated sub_volume 4dbd65ca-97ab-4433-af11-d4eebbebd5f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:21.011498854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 133 len 16384 data: 19 17 17 14 02/30 Read block 176 len 36864 03/30 Read block 376 len 20480 04/30 Flush 05/30 Write block 63 len 24576 data: 10 12 11 13 13 14 06/30 Write block 216 len 12288 data: 14 14 11 07/30 Flush 08/30 Write block 130 len 16384 data: 17 22 18 20 09/30 Write block 468 len 20480 data: 10 11 11 12 11 10/30 Read block 46 len 40960 11/30 Read block 599 len 4096 12/30 Write block 179 len 20480 data: 8 9 9 11 11 13/30 Write block 25 len 20480 data: 8 9 7 7 7 14/30 Write block 131 len 24576 data: 23 19 21 18 18 15 15/30 Read block 243 len 40960 16/30 Write block 191 len 28672 data: 17 14 11 12 16 15 15 17/30 Read block 288 len 16384 18/30 Flush 19/30 Write block 93 len 24576 data: 14 11 10 10 12 9 20/30 Flush 21/30 Write block 346 len 8192 data: 13 11 22/30 Read block 541 len 8192 23/30 Flush 24/30 Flush 25/30 Write block 2 len 32768 data: 8 11 10 14 13 10 14 13 26/30 Write block 522 len 4096 data: 7 27/30 Write block 119 len 24576 data: 11 11 12 12 12 15 28/30 Read block 421 len 12288 29/30 Read block 380 len 40960 30/30 Write block 186 len 8192 data: 15 15 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Read 9 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Read 10 Done Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 30 33 Done 29 28 0 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:21.023347747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1832} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 268 268 268 T T F <--- 1 020-039 156 156 147 272 272 254 F F F <--- 2 040-059 153 153 153 266 266 266 F F F 3 060-079 156 156 154 271 271 267 F F F <--- 4 080-099 156 156 151 273 273 262 F F F <--- 5 100-119 155 155 155 269 269 269 T T F <--- 6 120-139 156 156 154 272 272 267 T T F <--- 7 140-159 154 154 154 268 268 268 F F F 8 160-179 156 156 149 272 272 258 F F F <--- 9 180-199 156 156 154 272 272 267 T T F <--- 10 200-219 156 156 155 271 271 269 F F F <--- 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 152 152 265 265 265 F F F 13 260-279 155 155 155 269 269 269 F F F 14 280-299 155 155 155 269 269 269 F F F 15 300-319 155 155 155 269 269 269 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 156 156 152 274 274 265 F F F <--- 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 156 156 152 272 272 263 F F F <--- 24 480-499 155 155 155 269 269 269 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 155 155 155 269 269 269 T T F <--- 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 156, Max flush: 274 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 156 156 272 272 272 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 156 156 156 271 271 271 F F F 4 080-099 156 156 156 273 273 273 F F F 5 100-119 157 157 157 275 275 275 F F F 6 120-139 157 157 157 275 275 275 F F F 7 140-159 154 154 154 268 268 268 F F F 8 160-179 156 156 156 272 272 272 F F F 9 180-199 157 157 157 275 275 275 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 152 152 265 265 265 F F F 13 260-279 155 155 155 269 269 269 F F F 14 280-299 155 155 155 269 269 269 F F F 15 300-319 155 155 155 269 269 269 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 156 156 156 274 274 274 F F F 18 360-379 148 148 148 256 256 256 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 156 156 156 272 272 272 F F F 24 480-499 155 155 155 269 269 269 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 157 157 157 275 275 275 F F F 27 540-559 152 152 152 263 263 263 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 157, Max flush: 275 Begin loop 79 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.350898098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:25.361571105Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.36162508Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.371830365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372131682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372169409Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372184236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372202354Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"Crucible 5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2 has session id: e0aa7a08-f530-4c35-a3b0-cf87c06e6d35","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372569451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Upstairs opts: Upstairs UUID: 5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372603837Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Crucible stats registered with UUID: 5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372619534Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:25.37263281Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372789491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372813382Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"1","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372828139Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.37296359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.372981188Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373011142Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373024668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373039015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"1","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373052921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373069479Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373084165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373098272Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"1","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"ds_connection connected from Ok(127.0.0.1:33887)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373118861Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"1","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"ds_connection connected from Ok(127.0.0.1:36100)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373133578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"ds_connection connected from Ok(127.0.0.1:46361)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373240026Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"io task","client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373784114Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"1","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373878707Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.373918215Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374133023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374172881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[0]R flush_numbers[0..12]: [275, 272, 266, 271, 273, 275, 275, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374274237Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[0]R generation[0..12]: [157, 156, 153, 156, 156, 157, 157, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374291145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374307592Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[1]R flush_numbers[0..12]: [275, 272, 266, 271, 273, 275, 275, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374323439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[1]R generation[0..12]: [157, 156, 153, 156, 156, 157, 157, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374342488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374359286Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[2]R flush_numbers[0..12]: [275, 272, 266, 271, 273, 275, 275, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374375453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[2]R generation[0..12]: [157, 156, 153, 156, 156, 157, 157, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.37439117Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374407538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Max found gen is 158",Wait for a query_work_queue command to finish before sending IO "v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374426846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374441353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374454859Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374468415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"":"downstairs","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374482622Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374495178Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"0","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374509394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"1","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374526332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"client":"2","":"downstairs","session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374541148Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374556716Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2 is now active with session: e0aa7a08-f530-4c35-a3b0-cf87c06e6d35","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374571492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374584188Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840,"session_id":"e0aa7a08-f530-4c35-a3b0-cf87c06e6d35"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374597364Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} {"msg":"Activated sub_volume 5bc6b5d0-0652-4ba5-a7af-bc667ec65cc2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:25.374608849Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 551 len 32768 data: 15 14 16 14 13 12 10 9 03/30 Write block 257 len 40960 data: 13 12 10 10 11 12 12 9 7 10 04/30 Read block 203 len 24576 05/30 Read block 276 len 20480 06/30 Read block 543 len 24576 07/30 Read block 103 len 20480 08/30 Read block 237 len 36864 09/30 Read block 333 len 24576 10/30 Read block 55 len 12288 11/30 Read block 294 len 36864 12/30 Write block 549 len 8192 data: 14 18 13/30 Write block 119 len 36864 data: 12 12 13 13 13 16 15 16 19 14/30 Write block 248 len 24576 data: 16 18 16 14 18 18 15/30 Write block 547 len 24576 data: 11 13 15 19 16 15 16/30 Flush 17/30 Write block 365 len 40960 data: 11 9 10 13 15 14 12 12 11 13 18/30 Read block 158 len 4096 19/30 Read block 228 len 40960 20/30 Read block 468 len 8192 21/30 Read block 213 len 16384 22/30 Read block 524 len 20480 23/30 Write block 94 len 40960 data: 12 11 11 13 10 8 9 9 8 8 24/30 Read block 96 len 40960 25/30 Write block 135 len 12288 data: 19 16 11 26/30 Read block 176 len 20480 27/30 Write block 533 len 24576 data: 11 11 13 10 9 12 28/30 Read block 340 len 28672 29/30 Write block 72 len 40960 data: 9 10 12 14 13 14 13 12 13 8 30/30 Write block 256 len 40960 data: 16 14 13 11 11 12 13 13 10 8 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:25.387798773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1840} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 156 156 272 272 272 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 156 156 156 271 271 271 F T T <--- 4 080-099 156 156 156 273 273 273 F T T <--- 5 100-119 157 158 158 275 277 277 F T T <--- 6 120-139 157 158 158 275 277 277 F T T <--- 7 140-159 154 154 154 268 268 268 F F F 8 160-179 156 156 156 272 272 272 F F F 9 180-199 157 157 157 275 275 275 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 152 158 158 265 277 277 F T T <--- 13 260-279 155 158 158 269 277 277 F T T <--- 14 280-299 155 155 155 269 269 269 F F F 15 300-319 155 155 155 269 269 269 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 156 156 156 274 274 274 F F F 18 360-379 148 148 148 256 256 256 F T T <--- 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 156 156 156 272 272 272 F F F 24 480-499 155 155 155 269 269 269 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 157 157 157 275 275 275 F T T <--- 27 540-559 152 158 158 263 277 277 T F F <--- 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 158, Max flush: 277 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 156 156 272 272 272 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 159 159 159 278 278 278 F F F 5 100-119 159 159 159 278 278 278 F F F 6 120-139 159 159 159 278 278 278 F F F 7 140-159 154 154 154 268 268 268 F F F 8 160-179 156 156 156 272 272 272 F F F 9 180-199 157 157 157 275 275 275 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 159 159 159 278 278 278 F F F 14 280-299 155 155 155 269 269 269 F F F 15 300-319 155 155 155 269 269 269 F F F 16 320-339 147 147 147 254 254 254 F F F 17 340-359 156 156 156 274 274 274 F F F 18 360-379 159 159 159 278 278 278 F F F 19 380-399 148 148 148 256 256 256 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 152 152 152 264 264 264 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 156 156 156 272 272 272 F F F 24 480-499 155 155 155 269 269 269 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 159 159 159 278 278 278 F F F 27 540-559 158 158 158 277 277 277 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 159, Max flush: 278 Begin loop 80 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.746349062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758202367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.75851699Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758554817Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758567453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758585261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"Crucible b69cf223-6481-4c86-b083-dd81a8b1f108 has session id: 43cf4968-4ae3-4957-af03-4050a942eceb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758920263Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Upstairs opts: Upstairs UUID: b69cf223-6481-4c86-b083-dd81a8b1f108, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758949156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Crucible stats registered with UUID: b69cf223-6481-4c86-b083-dd81a8b1f108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.758964353Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:29.758977759Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759178581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759206624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759221961Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759342445Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759359133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"b69cf223-6481-4c86-b083-dd81a8b1f108 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759376621Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759389417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759403243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759417199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759430876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759444902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759459048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"ds_connection connected from Ok(127.0.0.1:47092)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759556133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"ds_connection connected from Ok(127.0.0.1:34459)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759693925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"ds_connection connected from Ok(127.0.0.1:53204)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.759721578Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"io task","client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760299511Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760349274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760391443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760676242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760717701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[0]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760826471Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[0]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760843609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760860226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[1]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760876434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[1]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760892391Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760908958Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[2]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 268, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760924906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[2]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 154, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760940513Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.76095676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:06:29.760970176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.760992677Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761005923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761019449Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"":"downstairs","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761033185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761046011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"0","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761059377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"1","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 level":30,"time":"2026-03-14T05:06:29.761075584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"client":"2","":"downstairs","session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761089741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761103317Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"b69cf223-6481-4c86-b083-dd81a8b1f108 is now active with session: 43cf4968-4ae3-4957-af03-4050a942eceb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761118014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761130889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848,"session_id":"43cf4968-4ae3-4957-af03-4050a942eceb"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761143825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} {"msg":"Activated sub_volume b69cf223-6481-4c86-b083-dd81a8b1f108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:29.761155301Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 398 len 8192 data: 14 14 02/30 Write block 465 len 8192 data: 12 11 03/30 Read block 506 len 24576 04/30 Read block 96 len 20480 05/30 Flush 06/30 Write block 299 len 32768 data: 6 6 6 8 8 6 8 9 07/30 Read block 92 len 28672 08/30 Write block 428 len 32768 data: 14 14 17 18 20 19 14 16 09/30 Read block 53 len 40960 10/30 Read block 287 len 24576 11/30 Write block 333 len 24576 data: 8 7 9 11 14 14 12/30 Read block 114 len 32768 13/30 Read block 150 len 4096 14/30 Flush 15/30 Write block 529 len 28672 data: 12 11 11 13 12 12 14 16/30 Read block 288 len 40960 17/30 Write block 376 len 16384 data: 10 14 16 19 18/30 Read block 518 len 36864 19/30 Write block 480 len 28672 data: 17 17 14 11 8 9 8 20/30 Read block 486 len 28672 21/30 Read block 487 len 28672 22/30 Write block 152 len 4096 data: 16 23/30 Read block 518 len 40960 24/30 Read block 505 len 16384 25/30 Read block 273 len 32768 26/30 Read block 222 len 28672 27/30 Read block 324 len 16384 28/30 Write block 276 len 8192 data: 16 15 29/30 Write block 542 len 24576 data: 12 12 13 12 12 12 30/30 Read block 450 len 24576 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 5 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 8 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:29.773901972Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1848} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 156 156 272 272 272 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 159 159 159 278 278 278 F F F 5 100-119 159 159 159 278 278 278 F F F 6 120-139 159 159 159 278 278 278 F F F 7 140-159 154 154 154 268 268 268 T T F <--- 8 160-179 156 156 156 272 272 272 F F F 9 180-199 157 157 157 275 275 275 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 159 159 159 278 278 278 T T F <--- 14 280-299 160 160 155 280 280 269 F F F <--- 15 300-319 160 160 155 280 280 269 F F F <--- 16 320-339 160 160 147 280 280 254 F F F <--- 17 340-359 156 156 156 274 274 274 F F F 18 360-379 159 159 159 278 278 278 T T F <--- 19 380-399 160 160 148 279 279 256 F F T <--- 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 152 280 280 264 F F F <--- 22 440-459 153 153 153 266 266 266 F F F 23 460-479 160 160 156 279 279 272 F F T <--- 24 480-499 155 155 155 269 269 269 T T F <--- 25 500-519 155 155 155 269 269 269 F F F 26 520-539 159 159 159 278 278 278 T T F <--- 27 540-559 158 158 158 277 277 277 T T F <--- 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 160, Max flush: 280 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 156 156 272 272 272 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 159 159 159 278 278 278 F F F 5 100-119 159 159 159 278 278 278 F F F 6 120-139 159 159 159 278 278 278 F F F 7 140-159 161 161 161 281 281 281 F F F 8 160-179 156 156 156 272 272 272 F F F 9 180-199 157 157 157 275 275 275 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 161 161 161 281 281 281 F F F 14 280-299 160 160 160 280 280 280 F F F 15 300-319 160 160 160 280 280 280 F F F 16 320-339 160 160 160 280 280 280 F F F 17 340-359 156 156 156 274 274 274 F F F 18 360-379 161 161 161 281 281 281 F F F 19 380-399 160 160 160 279 279 279 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 153 153 153 266 266 266 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 161 161 161 281 281 281 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 161 161 161 281 281 281 F F F 27 540-559 161 161 161 281 281 281 F F F 28 560-579 147 147 147 254 254 254 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 161, Max flush: 281 Begin loop 81 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.101082894Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:34.111721096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.111772099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122007768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122293988Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122331755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122343611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122361219Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"Crucible 9cae25ac-e486-42ef-a4cc-57a5b51d6806 has session id: 6d15fd72-f0f2-4267-8e67-df6c1bf5cfda","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122708627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Upstairs opts: Upstairs UUID: 9cae25ac-e486-42ef-a4cc-57a5b51d6806, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.12273744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Crucible stats registered with UUID: 9cae25ac-e486-42ef-a4cc-57a5b51d6806","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122752207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:34.122765443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122924735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122948156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.122962532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123098975Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123113651Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"9cae25ac-e486-42ef-a4cc-57a5b51d6806 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123130409Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123145326Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123159392Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123173418Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123191717Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123206173Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.12322047Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"ds_connection connected from Ok(127.0.0.1:42116)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123365766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"ds_connection connected from Ok(127.0.0.1:61390)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123451895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"ds_connection connected from Ok(127.0.0.1:51179)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.123479598Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"io task","client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124052729Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124105993Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124123581Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124419315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124445948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[0]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 281, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124511137Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[0]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 161, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124532317Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124548965Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[1]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 281, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124565052Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[1]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 161, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124580999Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124597337Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[2]R flush_numbers[0..12]: [275, 272, 266, 278, 278, 278, 278, 281, 272, 275, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124613034Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[2]R generation[0..12]: [157, 156, 153, 159, 159, 159, 159, 161, 156, 157, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124628821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124645028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Max found gen is 162","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:06:34.124664527Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124678844Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.12469223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124705506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"":"downstairs","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124719552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124732258Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"0","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124748635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"client":"1","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124762452Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 client":"2","":"downstairs","session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.12477982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124793206Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"9cae25ac-e486-42ef-a4cc-57a5b51d6806 is now active with session: 6d15fd72-f0f2-4267-8e67-df6c1bf5cfda","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124808032Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124820798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856,"session_id":"6d15fd72-f0f2-4267-8e67-df6c1bf5cfda"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.124834384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} {"msg":"Activated sub_volume 9cae25ac-e486-42ef-a4cc-57a5b51d6806","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:34.12484614Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 343 len 12288 data: 13 12 14 02/30 Write block 358 len 32768 data: 17 15 14 10 10 10 12 12 03/30 Write block 339 len 40960 data: 14 15 15 11 14 13 15 14 12 13 04/30 Write block 338 len 4096 data: 15 05/30 Read block 562 len 32768 06/30 Write block 567 len 36864 data: 12 14 17 19 17 17 13 16 19 07/30 Write block 180 len 4096 data: 10 08/30 Write block 26 len 24576 data: 10 8 8 8 8 10 09/30 Read block 472 len 16384 10/30 Read block 488 len 8192 11/30 Flush 12/30 Flush 13/30 Write block 274 len 8192 data: 14 14 14/30 Read block 37 len 12288 15/30 Write block 105 len 16384 data: 6 8 9 14 16/30 Write block 176 len 32768 data: 11 13 12 9 11 10 12 12 17/30 Read block 566 len 24576 18/30 Read block 407 len 32768 19/30 Read block 314 len 32768 20/30 Read block 356 len 8192 21/30 Read block 335 len 8192 22/30 Read block 521 len 24576 23/30 Write block 117 len 40960 data: 11 11 13 13 14 14 14 17 16 17 24/30 Read block 256 len 12288 25/30 Flush 26/30 Flush 27/30 Write block 446 len 32768 data: 11 8 12 12 12 10 11 11 28/30 Read block 341 len 12288 29/30 Write block 569 len 32768 data: 18 20 18 18 14 17 20 14 30/30 Read block 238 len 20480 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 8 Sent Done Done false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 3 30 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:34.137426576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1856} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 156 162 162 272 282 282 F F F <--- 2 040-059 153 153 153 266 266 266 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 159 159 159 278 278 278 F F F 5 100-119 159 162 162 278 284 284 F F F <--- 6 120-139 159 162 162 278 284 284 F F F <--- 7 140-159 161 161 161 281 281 281 F F F 8 160-179 156 162 162 272 284 284 F F F <--- 9 180-199 157 162 162 275 284 284 F F F <--- 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 161 162 162 281 284 284 F F F <--- 14 280-299 160 160 160 280 280 280 F F F 15 300-319 160 160 160 280 280 280 F F F 16 320-339 160 162 162 280 282 282 T F F <--- 17 340-359 156 162 162 274 282 282 T F F <--- 18 360-379 161 162 162 281 282 282 T F F <--- 19 380-399 160 160 160 279 279 279 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 153 153 153 266 266 266 F T T <--- 23 460-479 160 160 160 279 279 279 F F F 24 480-499 161 161 161 281 281 281 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 161 161 161 281 281 281 F F F 27 540-559 161 161 161 281 281 281 F F F 28 560-579 147 162 162 254 282 282 F T T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 162, Max flush: 284 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 162 162 162 282 282 282 F F F 2 040-059 153 153 153 266 266 266 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 159 159 159 278 278 278 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 162 162 162 284 284 284 F F F 7 140-159 161 161 161 281 281 281 F F F 8 160-179 162 162 162 284 284 284 F F F 9 180-199 162 162 162 284 284 284 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 162 162 162 284 284 284 F F F 14 280-299 160 160 160 280 280 280 F F F 15 300-319 160 160 160 280 280 280 F F F 16 320-339 162 162 162 282 282 282 F F F 17 340-359 162 162 162 282 282 282 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 160 160 160 279 279 279 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 163 163 163 285 285 285 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 161 161 161 281 281 281 F F F 25 500-519 155 155 155 269 269 269 F F F 26 520-539 161 161 161 281 281 281 F F F 27 540-559 161 161 161 281 281 281 F F F 28 560-579 163 163 163 285 285 285 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 163, Max flush: 285 Begin loop 82 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.4691395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.480951366Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481281576Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481320474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48133351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481351778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"Crucible 67716ffe-a5d9-4f4b-a158-2d1265a3eb73 has session id: 3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481731881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Upstairs opts: Upstairs UUID: 67716ffe-a5d9-4f4b-a158-2d1265a3eb73, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481765847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Crucible stats registered with UUID: 67716ffe-a5d9-4f4b-a158-2d1265a3eb73","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481785135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:38.481798572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481931962Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481960295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.481975192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48212572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482142528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"67716ffe-a5d9-4f4b-a158-2d1265a3eb73 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482159606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482172382Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482186658Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482200795Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482214591Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482228827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482242684Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"ds_connection connected from Ok(127.0.0.1:62593)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48225743Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"ds_connection connected from Ok(127.0.0.1:64761)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482275359Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"ds_connection connected from Ok(127.0.0.1:64779)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.482290515Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"io task","client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483057565Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483117462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48313502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483418409Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483460878Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[0]R flush_numbers[0..12]: [275, 282, 266, 278, 278, 284, 284, 281, 284, 284, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483525217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[0]R generation[0..12]: [157, 162, 153, 159, 159, 162, 162, 161, 162, 162, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483541885Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483558383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[1]R flush_numbers[0..12]: [275, 282, 266, 278, 278, 284, 284, 281, 284, 284, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48357399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[1]R generation[0..12]: [157, 162, 153, 159, 159, 162, 162, 161, 162, 162, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483589547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483605854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[2]R flush_numbers[0..12]: [275, 282, 266, 278, 278, 284, 284, 281, 284, 284, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483621241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[2]R generation[0..12]: [157, 162, 153, 159, 159, 162, 162, 161, 162, 162, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.48364014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483656777Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":Wait for a query_work_queue command to finish before sending IO "Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483675696Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483689742Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Next flush: 286","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483703068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483716545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"":"downstairs","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483730941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483743617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"0","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483757463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"1","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483774351Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"client":"2","":"downstairs","session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483789058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483802634Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"67716ffe-a5d9-4f4b-a158-2d1265a3eb73 is now active with session: 3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483817671Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483830546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864,"session_id":"3853d02e-8e7b-41f3-8c29-0c5dcb4dbd96"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483843722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} {"msg":"Activated sub_volume 67716ffe-a5d9-4f4b-a158-2d1265a3eb73","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:38.483854998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 150 len 12288 data: 15 16 17 02/30 Write block 390 len 36864 data: 14 14 15 16 16 12 14 15 15 03/30 Read block 70 len 36864 04/30 Write block 146 len 16384 data: 15 14 16 16 05/30 Read block 36 len 32768 06/30 Write block 165 len 20480 data: 11 14 15 14 15 07/30 Read block 413 len 12288 08/30 Read block 521 len 40960 09/30 Write block 382 len 24576 data: 21 23 20 17 16 16 10/30 Read block 290 len 36864 11/30 Read block 253 len 4096 12/30 Write block 175 len 4096 data: 12 13/30 Write block 84 len 4096 data: 11 14/30 Flush 15/30 Write block 285 len 40960 data: 11 13 14 15 18 17 16 15 14 12 16/30 Read block 360 len 40960 17/30 Read block 391 len 36864 18/30 Write block 38 len 12288 data: 9 8 9 19/30 Read block 33 len 20480 20/30 Write block 260 len 32768 data: 12 13 14 14 11 9 11 11 21/30 Write block 187 len 24576 data: 16 15 18 15 18 15 22/30 Flush 23/30 Write block 128 len 32768 data: 18 17 18 24 20 22 19 20 24/30 Write block 342 len 20480 data: 12 15 14 16 15 25/30 Write block 31 len 32768 data: 11 9 10 13 11 10 9 10 26/30 Read block 84 len 36864 27/30 Write block 318 len 12288 data: 20 21 22 28/30 Write block 131 len 12288 data: 25 21 23 29/30 Write block 270 len 28672 data: 13 14 15 15 15 15 17 30/30 Write block 494 len 36864 data: 17 13 14 15 17 15 14 13 12 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:38.49601158Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1864} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 164 162 164 287 282 287 T F T <--- 2 040-059 164 153 164 287 266 287 F F F <--- 3 060-079 159 159 159 278 278 278 F F F 4 080-099 164 159 164 286 278 286 F F F <--- 5 100-119 162 162 162 284 284 284 F F F 6 120-139 162 162 162 284 284 284 T F T <--- 7 140-159 164 161 164 286 281 286 F T F <--- 8 160-179 164 162 164 286 284 286 F F F <--- 9 180-199 164 162 164 287 284 287 F F F <--- 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 164 162 164 287 284 287 T F T <--- 14 280-299 164 160 164 287 280 287 F F F <--- 15 300-319 160 160 160 280 280 280 T F T <--- 16 320-339 162 162 162 282 282 282 T F T <--- 17 340-359 162 162 162 282 282 282 T F T <--- 18 360-379 162 162 162 282 282 282 F F F 19 380-399 164 160 164 286 279 286 F T F <--- 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 163 163 163 285 285 285 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 161 161 161 281 281 281 T F T <--- 25 500-519 155 155 155 269 269 269 T F T <--- 26 520-539 161 161 161 281 281 281 F F F 27 540-559 161 161 161 281 281 281 F F F 28 560-579 163 163 163 285 285 285 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 164, Max flush: 287 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 165 165 165 288 288 288 F F F 2 040-059 164 164 164 287 287 287 F F F 3 060-079 159 159 159 278 278 278 F F F 4 080-099 164 164 164 286 286 286 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 165 165 165 288 288 288 F F F 7 140-159 164 164 164 286 286 286 F F F 8 160-179 164 164 164 286 286 286 F F F 9 180-199 164 164 164 287 287 287 F F F 10 200-219 156 156 156 271 271 271 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 164 164 164 287 287 287 F F F 15 300-319 165 165 165 288 288 288 F F F 16 320-339 165 165 165 288 288 288 F F F 17 340-359 165 165 165 288 288 288 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 150 150 150 259 259 259 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 163 163 163 285 285 285 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 165 165 165 288 288 288 F F F 25 500-519 165 165 165 288 288 288 F F F 26 520-539 161 161 161 281 281 281 F F F 27 540-559 161 161 161 281 281 281 F F F 28 560-579 163 163 163 285 285 285 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 165, Max flush: 288 Begin loop 83 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.83217616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.843890051Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84423847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844278288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844290904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844308642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"Crucible 3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f has session id: 5cd04fc6-6ff2-4625-99de-a11ea5bdacdb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844675159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Upstairs opts: Upstairs UUID: 3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844701631Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Crucible stats registered with UUID: 3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844716818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:42.844730004Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844887725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844910846Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.844925603Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845058063Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84507338Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845091278Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845104214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84511818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845132407Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845145903Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84518436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845212823Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"ds_connection connected from Ok(127.0.0.1:34956)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845238835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"ds_connection connected from Ok(127.0.0.1:51450)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845254492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"ds_connection connected from Ok(127.0.0.1:56615)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.845269209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"io task","client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84606322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846099917Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846117335Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846445585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846462992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[0]R flush_numbers[0..12]: [275, 288, 287, 278, 286, 284, 288, 286, 286, 287, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.8464801Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[0]R generation[0..12]: [157, 165, 164, 159, 164, 162, 165, 164, 164, 164, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846496178Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846512485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[1]R flush_numbers[0..12]: [275, 288, 287, 278, 286, 284, 288, 286, 286, 287, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846528252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[1]R generation[0..12]: [157, 165, 164, 159, 164, 162, 165, 164, 164, 164, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84654451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846560907Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[2]R flush_numbers[0..12]: [275, 288, 287, 278, 286, 284, 288, 286, 286, 287, 271, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846576554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[2]R generation[0..12]: [157, 165, 164, 159, 164, 162, 165, 164, 164, 164, 156, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846592011Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846608399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846621955Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Generation requested: 166 >= found:166",Wait for a query_work_queue command to finish before sending IO "v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846642464Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Next flush: 289","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84665612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846669546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"":"downstairs","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846685814Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84669849Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"0","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846712456Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"1","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846726352Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"client":"2","":"downstairs","session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846740549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846756816Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f is now active with session: 5cd04fc6-6ff2-4625-99de-a11ea5bdacdb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846772003Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846784649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872,"session_id":"5cd04fc6-6ff2-4625-99de-a11ea5bdacdb"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.846798325Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} {"msg":"Activated sub_volume 3c762259-4ced-4fd9-b8fe-bdd40c1b8a7f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:42.84680975Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 406 len 8192 data: 8 12 02/30 Read block 11 len 40960 03/30 Read block 53 len 28672 04/30 Read block 295 len 32768 05/30 Write block 138 len 12288 data: 13 13 14 06/30 Write block 65 len 28672 data: 12 14 14 15 15 10 9 07/30 Read block 197 len 4096 08/30 Write block 408 len 8192 data: 12 15 09/30 Flush 10/30 Write block 131 len 32768 data: 26 22 24 20 21 17 12 14 11/30 Flush 12/30 Write block 339 len 36864 data: 15 16 16 13 16 15 17 16 13 13/30 Write block 349 len 4096 data: 15 14/30 Write block 552 len 28672 data: 16 17 15 14 13 11 10 15/30 Write block 387 len 40960 data: 17 15 15 15 15 16 17 17 13 15 16/30 Read block 285 len 40960 17/30 Read block 486 len 12288 18/30 Write block 501 len 28672 data: 14 13 13 13 15 13 13 19/30 Read block 544 len 40960 20/30 Read block 436 len 32768 21/30 Read block 337 len 40960 22/30 Write block 291 len 32768 data: 17 16 15 13 7 7 6 5 23/30 Write block 326 len 32768 data: 16 13 12 11 10 8 8 9 24/30 Write block 128 len 36864 data: 19 18 19 27 23 25 21 22 18 25/30 Flush 26/30 Write block 204 len 16384 data: 7 8 10 9 27/30 Read block 68 len 4096 28/30 Write block 90 len 4096 data: 18 29/30 Read block 386 len 40960 30/30 Write block 488 len 28672 data: 11 15 16 15 14 15 18 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 1 Done Done Done false 1007 Acked Write 2 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 29 29 7 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:42.85935395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1872} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 165 165 165 288 288 288 F F F 2 040-059 164 164 164 287 287 287 F F F 3 060-079 166 166 159 289 289 278 F F T <--- 4 080-099 164 164 164 286 286 286 T T F <--- 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 165 291 291 288 F F T <--- 7 140-159 166 166 164 289 289 286 F F T <--- 8 160-179 164 164 164 286 286 286 F F F 9 180-199 164 164 164 287 287 287 F F F 10 200-219 156 156 156 271 271 271 T T F <--- 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 166 166 164 291 291 287 F F F <--- 15 300-319 165 165 165 288 288 288 F F F 16 320-339 166 166 165 291 291 288 F F F <--- 17 340-359 166 166 165 291 291 288 F F F <--- 18 360-379 162 162 162 282 282 282 F F F 19 380-399 166 166 164 291 291 286 F F F <--- 20 400-419 166 166 150 289 289 259 F F T <--- 21 420-439 160 160 160 280 280 280 F F F 22 440-459 163 163 163 285 285 285 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 165 165 165 288 288 288 T T F <--- 25 500-519 166 166 165 291 291 288 F F F <--- 26 520-539 161 161 161 281 281 281 F F F 27 540-559 166 166 161 291 291 281 F F F <--- 28 560-579 163 163 163 285 285 285 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 166, Max flush: 291 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 165 165 165 288 288 288 F F F 2 040-059 164 164 164 287 287 287 F F F 3 060-079 166 166 166 289 289 289 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 166 166 166 289 289 289 F F F 8 160-179 164 164 164 286 286 286 F F F 9 180-199 164 164 164 287 287 287 F F F 10 200-219 167 167 167 292 292 292 F F F 11 220-239 147 147 147 254 254 254 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 166 166 166 291 291 291 F F F 15 300-319 165 165 165 288 288 288 F F F 16 320-339 166 166 166 291 291 291 F F F 17 340-359 166 166 166 291 291 291 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 166 166 166 291 291 291 F F F 20 400-419 166 166 166 289 289 289 F F F 21 420-439 160 160 160 280 280 280 F F F 22 440-459 163 163 163 285 285 285 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 167 167 167 292 292 292 F F F 25 500-519 166 166 166 291 291 291 F F F 26 520-539 161 161 161 281 281 281 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 163 163 163 285 285 285 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 167, Max flush: 292 Begin loop 84 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.188649434Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.200293984Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.200603315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.200641302Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.200653498Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.200671256Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"Crucible 8469d2e2-eaaa-4cb1-8501-548ae4185b50 has session id: a1af117b-0125-4927-9154-c9c94e60dae4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201000236Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Upstairs opts: Upstairs UUID: 8469d2e2-eaaa-4cb1-8501-548ae4185b50, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201041324Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Crucible stats registered with UUID: 8469d2e2-eaaa-4cb1-8501-548ae4185b50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201056991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:47.201070688Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201266937Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201298281Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201313538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201420227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201435734Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"8469d2e2-eaaa-4cb1-8501-548ae4185b50 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201466828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201480594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201494761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201508617Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201522033Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.20155876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201584422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"ds_connection connected from Ok(127.0.0.1:43140)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201607172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"ds_connection connected from Ok(127.0.0.1:59288)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201622589Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"ds_connection connected from Ok(127.0.0.1:60663)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.201637986Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"io task","client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202348489Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202379713Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202397121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.20270145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202733505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[0]R flush_numbers[0..12]: [275, 288, 287, 289, 292, 284, 291, 289, 286, 287, 292, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202832039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[0]R generation[0..12]: [157, 165, 164, 166, 167, 162, 166, 166, 164, 164, 167, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202848807Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202865154Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[1]R flush_numbers[0..12]: [275, 288, 287, 289, 292, 284, 291, 289, 286, 287, 292, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202881302Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[1]R generation[0..12]: [157, 165, 164, 166, 167, 162, 166, 166, 164, 164, 167, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202897049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202915948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[2]R flush_numbers[0..12]: [275, 288, 287, 289, 292, 284, 291, 289, 286, 287, 292, 254]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202931785Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[2]R generation[0..12]: [157, 165, 164, 166, 167, 162, 166, 166, 164, 164, 167, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202947612Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202964109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:06:47.202977616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.202996104Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Next flush: 293","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.20300948Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203022676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"":"downstairs","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203037163Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203049628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"0","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203063985Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"1","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:47.203077761Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"client":"2","":"downstairs","session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203094009Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203107535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"8469d2e2-eaaa-4cb1-8501-548ae4185b50 is now active with session: a1af117b-0125-4927-9154-c9c94e60dae4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203122672Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203136898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880,"session_id":"a1af117b-0125-4927-9154-c9c94e60dae4"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203150234Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} {"msg":"Activated sub_volume 8469d2e2-eaaa-4cb1-8501-548ae4185b50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:47.203161809Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 321 len 36864 data: 19 17 16 15 14 17 14 13 12 02/30 Flush 03/30 Write block 417 len 32768 data: 9 7 8 10 10 13 15 14 04/30 Read block 41 len 36864 05/30 Read block 525 len 40960 06/30 Write block 319 len 40960 data: 22 23 20 18 17 16 15 18 15 14 07/30 Write block 38 len 20480 data: 11 9 10 7 9 08/30 Read block 371 len 32768 09/30 Read block 233 len 28672 10/30 Write block 203 len 20480 data: 8 8 9 11 10 11/30 Write block 447 len 32768 data: 9 13 13 13 11 12 12 10 12/30 Read block 150 len 20480 13/30 Write block 150 len 24576 data: 16 17 18 12 12 11 14/30 Read block 410 len 28672 15/30 Read block 576 len 4096 16/30 Read block 454 len 24576 17/30 Write block 183 len 24576 data: 13 12 14 16 17 16 18/30 Flush 19/30 Write block 516 len 32768 data: 11 11 8 7 7 6 8 8 20/30 Flush 21/30 Write block 237 len 12288 data: 12 10 6 22/30 Write block 574 len 8192 data: 18 21 23/30 Write block 420 len 32768 data: 11 11 14 16 15 17 14 13 24/30 Write block 400 len 20480 data: 11 10 9 6 6 25/30 Read block 458 len 32768 26/30 Read block 262 len 16384 27/30 Write block 315 len 40960 data: 14 17 17 21 23 24 21 19 18 17 28/30 Write block 37 len 4096 data: 10 29/30 Read block 376 len 16384 30/30 Write block 397 len 16384 data: 16 16 15 12 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 8 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 27 0 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1001)) Some(JobId(1019)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:47.216694399Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1880} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 168 165 168 294 288 294 T F T <--- 2 040-059 168 164 168 294 287 294 F F F <--- 3 060-079 166 166 166 289 289 289 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 168 166 168 294 289 294 F F F <--- 8 160-179 164 164 164 286 286 286 F F F 9 180-199 168 164 168 294 287 294 F F F <--- 10 200-219 168 167 168 294 292 294 F F F <--- 11 220-239 147 147 147 254 254 254 T F T <--- 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 166 166 166 291 291 291 F F F 15 300-319 168 165 168 294 288 294 T F T <--- 16 320-339 168 168 168 294 293 294 T F T <--- 17 340-359 166 166 166 291 291 291 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 166 166 166 291 291 291 T F T <--- 20 400-419 168 166 168 294 289 294 T F T <--- 21 420-439 168 160 168 294 280 294 T F T <--- 22 440-459 168 163 168 294 285 294 F F F <--- 23 460-479 160 160 160 279 279 279 F F F 24 480-499 167 167 167 292 292 292 F F F 25 500-519 168 166 168 295 291 295 F F F <--- 26 520-539 168 161 168 295 281 295 F F F <--- 27 540-559 166 166 166 291 291 291 F F F 28 560-579 163 163 163 285 285 285 T F T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 168, Max flush: 295 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 F F F 1 020-039 169 169 169 296 296 296 F F F 2 040-059 168 168 168 294 294 294 F F F 3 060-079 166 166 166 289 289 289 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 168 168 168 294 294 294 F F F 8 160-179 164 164 164 286 286 286 F F F 9 180-199 168 168 168 294 294 294 F F F 10 200-219 168 168 168 294 294 294 F F F 11 220-239 169 169 169 296 296 296 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 166 166 166 291 291 291 F F F 15 300-319 169 169 169 296 296 296 F F F 16 320-339 169 169 169 296 296 296 F F F 17 340-359 166 166 166 291 291 291 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 169 169 169 296 296 296 F F F 21 420-439 169 169 169 296 296 296 F F F 22 440-459 168 168 168 294 294 294 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 167 167 167 292 292 292 F F F 25 500-519 168 168 168 295 295 295 F F F 26 520-539 168 168 168 295 295 295 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 169 169 169 296 296 296 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 169, Max flush: 296 Begin loop 85 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.5528804Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.564535574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.564851658Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.564890936Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.564903312Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56492133Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"Crucible 945494e5-1f6d-4756-844c-589377283d0e has session id: b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565278533Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Upstairs opts: Upstairs UUID: 945494e5-1f6d-4756-844c-589377283d0e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565313799Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Crucible stats registered with UUID: 945494e5-1f6d-4756-844c-589377283d0e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565330176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:51.565343572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56549274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565518812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565533889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565666659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565685218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"945494e5-1f6d-4756-844c-589377283d0e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565702926Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565716062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565730098Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565744095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565774288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565789055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565803331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:53213)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56582155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61865)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.565836627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:35858)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56593176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"io task","client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.566602425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.566631949Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.566660442Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.566961539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567012932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[0]R flush_numbers[0..12]: [275, 296, 294, 289, 292, 284, 291, 294, 286, 294, 294, 296]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567086375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[0]R generation[0..12]: [157, 169, 168, 166, 167, 162, 166, 168, 164, 168, 168, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567103383Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567119881Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[1]R flush_numbers[0..12]: [275, 296, 294, 289, 292, 284, 291, 294, 286, 294, 294, 296]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567135678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[1]R generation[0..12]: [157, 169, 168, 166, 167, 162, 166, 168, 164, 168, 168, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567151165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567167532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[2]R flush_numbers[0..12]: [275, 296, 294, 289, 292, 284, 291, 294, 286, 294, 294, 296]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56718343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[2]R generation[0..12]: [157, 169, 168, 166, 167, 162, 166, 168, 164, 168, 168, 169]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567199377Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567215564Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56722911Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56724951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Next flush: 297","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567265187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567278643Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"":"downstairs","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56729327Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567305975Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"0","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567320122Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"1","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:51.567333998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"client":"2","":"downstairs","session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567349295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567362831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"945494e5-1f6d-4756-844c-589377283d0e is now active with session: b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567377778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567390644Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888,"session_id":"b4f1ab6d-e5b9-4451-9c99-ebfbe76eb84a"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.56740411Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} {"msg":"Activated sub_volume 945494e5-1f6d-4756-844c-589377283d0e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:51.567415785Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 482 len 8192 data: 15 12 03/30 Read block 70 len 4096 04/30 Write block 64 len 40960 data: 13 13 15 15 16 16 11 10 10 11 05/30 Write block 147 len 32768 data: 15 17 17 17 18 19 13 13 06/30 Read block 197 len 32768 07/30 Write block 19 len 28672 data: 4 3 3 5 6 8 9 08/30 Read block 472 len 20480 09/30 Write block 46 len 16384 data: 12 13 15 13 10/30 Read block 107 len 12288 11/30 Write block 513 len 12288 data: 12 15 11 12/30 Write block 220 len 28672 data: 15 13 11 13 13 15 16 13/30 Read block 1 len 40960 14/30 Read block 528 len 16384 15/30 Write block 183 len 36864 data: 14 13 15 17 18 17 19 16 19 16/30 Read block 316 len 8192 17/30 Read block 378 len 24576 18/30 Write block 404 len 4096 data: 7 19/30 Write block 235 len 12288 data: 14 14 13 20/30 Write block 522 len 20480 data: 9 9 8 10 7 21/30 Read block 513 len 36864 22/30 Write block 284 len 28672 data: 10 12 14 15 16 19 18 23/30 Write block 302 len 36864 data: 9 9 7 9 10 12 13 11 12 24/30 Read block 98 len 16384 25/30 Write block 519 len 16384 data: 8 8 7 10 26/30 Read block 489 len 12288 27/30 Read block 543 len 8192 28/30 Read block 409 len 36864 29/30 Read block 165 len 4096 30/30 Read block 35 len 32768 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Sent Done false 1002 Acked Read 1 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:51.579419909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1888} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 275 275 275 T F T <--- 1 020-039 169 169 169 296 296 296 T F T <--- 2 040-059 168 168 168 294 294 294 T F T <--- 3 060-079 166 166 166 289 289 289 T F T <--- 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 168 168 168 294 294 294 T F T <--- 8 160-179 164 164 164 286 286 286 F F F 9 180-199 168 168 168 294 294 294 T F T <--- 10 200-219 168 168 168 294 294 294 F F F 11 220-239 169 169 169 296 296 296 T F T <--- 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 166 166 166 291 291 291 T F T <--- 15 300-319 169 169 169 296 296 296 T F T <--- 16 320-339 169 169 169 296 296 296 F F F 17 340-359 166 166 166 291 291 291 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 169 169 169 296 296 296 T F T <--- 21 420-439 169 169 169 296 296 296 F F F 22 440-459 168 168 168 294 294 294 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 167 167 167 292 292 292 T F T <--- 25 500-519 168 168 168 295 295 295 T F T <--- 26 520-539 168 168 168 295 295 295 T F T <--- 27 540-559 166 166 166 291 291 291 F F F 28 560-579 169 169 169 296 296 296 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 169, Max flush: 296 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 297 297 297 F F F 1 020-039 170 170 170 297 297 297 F F F 2 040-059 170 170 170 297 297 297 F F F 3 060-079 170 170 170 297 297 297 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 170 170 170 297 297 297 F F F 8 160-179 164 164 164 286 286 286 F F F 9 180-199 170 170 170 297 297 297 F F F 10 200-219 168 168 168 294 294 294 F F F 11 220-239 170 170 170 297 297 297 F F F 12 240-259 159 159 159 278 278 278 F F F 13 260-279 165 165 165 288 288 288 F F F 14 280-299 170 170 170 297 297 297 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 169 169 169 296 296 296 F F F 17 340-359 166 166 166 291 291 291 F F F 18 360-379 162 162 162 282 282 282 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 170 170 170 297 297 297 F F F 21 420-439 169 169 169 296 296 296 F F F 22 440-459 168 168 168 294 294 294 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 170 170 170 297 297 297 F F F 25 500-519 170 170 170 297 297 297 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 169 169 169 296 296 296 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 170, Max flush: 297 Begin loop 86 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.912377741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:55.923005798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.923061243Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.933474012Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.933767076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.933808484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93382097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.933839258Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"Crucible 5f331d81-5e05-400e-b328-3054f0cfdea4 has session id: acb789ba-f5b7-411a-a840-7a8d9005f0c3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934180454Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Upstairs opts: Upstairs UUID: 5f331d81-5e05-400e-b328-3054f0cfdea4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93421509Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Crucible stats registered with UUID: 5f331d81-5e05-400e-b328-3054f0cfdea4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934230857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:55.934244073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934399583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934424075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934438921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934561907Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934577064Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"5f331d81-5e05-400e-b328-3054f0cfdea4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934592821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934605737Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934622025Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934636161Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934649767Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934664214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93467824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:42199)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934738618Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:39029)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.934766561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48775)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93485296Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"io task","client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935411854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935438116Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935454774Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935808765Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935834156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[0]R flush_numbers[0..12]: [297, 297, 297, 297, 292, 284, 291, 297, 286, 297, 294, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935877596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[0]R generation[0..12]: [170, 170, 170, 170, 167, 162, 166, 170, 164, 170, 168, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935894934Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935915013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[1]R flush_numbers[0..12]: [297, 297, 297, 297, 292, 284, 291, 297, 286, 297, 294, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93593077Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[1]R generation[0..12]: [170, 170, 170, 170, 167, 162, 166, 170, 164, 170, 168, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935946828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935963445Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[2]R flush_numbers[0..12]: [297, 297, 297, 297, 292, 284, 291, 297, 286, 297, 294, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.935979242Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[2]R generation[0..12]: [170, 170, 170, 170, 167, 162, 166, 170, 164, 170, 168, 170]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93599481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936011407Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Max found gen is 171","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:06:55.936030826Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Generation requested: 172 >= found:171","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936045092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Next flush: 298","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936058288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936071524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"":"downstairs","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936085491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936098117Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"0","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936112073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"1","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.93612817Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"client":"2","":"downstairs","session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936145478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936159104Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"5f331d81-5e05-400e-b328-3054f0cfdea4 is now active with session: acb789ba-f5b7-411a-a840-7a8d9005f0c3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936174211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936186867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896,"session_id":"acb789ba-f5b7-411a-a840-7a8d9005f0c3"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936200223Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} {"msg":"Activated sub_volume 5f331d81-5e05-400e-b328-3054f0cfdea4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:55.936211628Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 47 len 12288 data: 14 16 14 02/30 Write block 229 len 16384 data: 15 15 18 17 03/30 Read block 139 len 24576 04/30 Flush 05/30 Write block 349 len 4096 data: 16 06/30 Write block 516 len 4096 data: 12 07/30 Flush 08/30 Write block 498 len 4096 data: 18 09/30 Write block 168 len 36864 data: 15 16 16 17 18 17 14 13 12 10/30 Write block 574 len 12288 data: 19 22 15 11/30 Read block 243 len 16384 12/30 Write block 330 len 8192 data: 11 9 13/30 Read block 497 len 28672 14/30 Write block 562 len 40960 data: 10 9 12 12 14 13 15 19 21 19 15/30 Read block 387 len 12288 16/30 Write block 57 len 40960 data: 9 10 10 10 8 10 11 14 14 16 17/30 Write block 372 len 24576 data: 13 12 14 12 11 15 18/30 Read block 550 len 16384 19/30 Read block 470 len 24576 20/30 Write block 432 len 36864 data: 21 20 15 17 19 19 20 18 16 21/30 Read block 369 len 4096 22/30 Read block 410 len 4096 23/30 Read block 39 len 16384 24/30 Read block 503 len 20480 25/30 Read block 59 len 8192 26/30 Write block 198 len 28672 data: 12 12 12 9 9 9 9 27/30 Write block 405 len 20480 data: 6 9 13 13 16 28/30 Write block 252 len 36864 data: 19 19 21 19 17 15 14 12 13 29/30 Read block 537 len 28672 30/30 Read block 462 len 16384 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Write 7 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:55.948010548Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1896} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 297 297 297 F F F 1 020-039 170 170 170 297 297 297 F F F 2 040-059 170 172 172 297 298 298 T T T <--- 3 060-079 170 170 170 297 297 297 F T T <--- 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 170 170 170 297 297 297 F F F 8 160-179 164 164 164 286 286 286 F T T <--- 9 180-199 170 170 170 297 297 297 F T T <--- 10 200-219 168 168 168 294 294 294 F T T <--- 11 220-239 170 172 172 297 298 298 F F F <--- 12 240-259 159 159 159 278 278 278 F T T <--- 13 260-279 165 165 165 288 288 288 F T T <--- 14 280-299 170 170 170 297 297 297 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 169 169 169 296 296 296 F T T <--- 17 340-359 166 172 172 291 299 299 F F F <--- 18 360-379 162 162 162 282 282 282 F T T <--- 19 380-399 169 169 169 296 296 296 F F F 20 400-419 170 170 170 297 297 297 F T T <--- 21 420-439 169 169 169 296 296 296 F T T <--- 22 440-459 168 168 168 294 294 294 F T T <--- 23 460-479 160 160 160 279 279 279 F F F 24 480-499 170 170 170 297 297 297 F T T <--- 25 500-519 170 172 172 297 299 299 F F F <--- 26 520-539 170 170 170 297 297 297 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 169 169 169 296 296 296 F T T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 172, Max flush: 299 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 297 297 297 F F F 1 020-039 170 170 170 297 297 297 F F F 2 040-059 173 173 173 300 300 300 F F F 3 060-079 173 173 173 300 300 300 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 170 170 170 297 297 297 F F F 8 160-179 173 173 173 300 300 300 F F F 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 172 172 172 298 298 298 F F F 12 240-259 173 173 173 300 300 300 F F F 13 260-279 173 173 173 300 300 300 F F F 14 280-299 170 170 170 297 297 297 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 173 173 173 300 300 300 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 173 173 173 300 300 300 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 173 173 173 300 300 300 F F F 21 420-439 173 173 173 300 300 300 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 173 173 173 300 300 300 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 173 173 173 300 300 300 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 173, Max flush: 300 Begin loop 87 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.300979778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:00.311587956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.311633207Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.321909794Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322199126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322238994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.32225129Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322269288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"Crucible 7730831c-989e-4173-806c-9f05cc725320 has session id: ee507bea-1829-436d-a66a-fcba9f789eac","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322608982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Upstairs opts: Upstairs UUID: 7730831c-989e-4173-806c-9f05cc725320, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322638156Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Crucible stats registered with UUID: 7730831c-989e-4173-806c-9f05cc725320","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322653123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:00.322666499Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322833725Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322856595Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.322871482Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"1","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.32302189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323038608Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"7730831c-989e-4173-806c-9f05cc725320 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323061388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323073904Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323088141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"1","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323101977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323115653Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.32313002Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323144726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"1","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"ds_connection connected from Ok(127.0.0.1:47642)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323206484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"1","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"ds_connection connected from Ok(127.0.0.1:60112)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323347909Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"ds_connection connected from Ok(127.0.0.1:44491)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323375811Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"io task","client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323959417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.323998785Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"1","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324017123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324308376Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324334798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[0]R flush_numbers[0..12]: [297, 297, 300, 300, 292, 284, 291, 297, 300, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324400378Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[0]R generation[0..12]: [170, 170, 173, 173, 167, 162, 166, 170, 173, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324418726Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324435424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[1]R flush_numbers[0..12]: [297, 297, 300, 300, 292, 284, 291, 297, 300, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324451461Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[1]R generation[0..12]: [170, 170, 173, 173, 167, 162, 166, 170, 173, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324467158Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324483786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[2]R flush_numbers[0..12]: [297, 297, 300, 300, 292, 284, 291, 297, 300, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324503205Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[2]R generation[0..12]: [170, 170, 173, 173, 167, 162, 166, 170, 173, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324518812Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324535039Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324555008Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324569425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Next flush: 301","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324582851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324596057Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"":"downstairs","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324610193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324623029Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"0","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324637176Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"1","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324653773Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"client":"2","":"downstairs","session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.32466842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324682246Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"7730831c-989e-4173-806c-9f05cc725320 is now active with session: ee507bea-1829-436d-a66a-fcba9f789eac","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324697673Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324710109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904,"session_id":"ee507bea-1829-436d-a66a-fcba9f789eac"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324725356Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} {"msg":"Activated sub_volume 7730831c-989e-4173-806c-9f05cc725320","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:00.324737061Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 1 len 4096 data: 7 02/30 Read block 426 len 24576 03/30 Read block 5 len 12288 04/30 Read block 243 len 12288 05/30 Read block 348 len 40960 06/30 Write block 546 len 40960 data: 13 13 14 16 20 17 17 18 16 15 07/30 Flush 08/30 Write block 564 len 36864 data: 13 13 15 14 16 20 22 20 19 09/30 Write block 325 len 8192 data: 16 19 10/30 Write block 547 len 36864 data: 14 15 17 21 18 18 19 17 16 11/30 Write block 170 len 36864 data: 17 18 19 18 15 14 13 14 13 12/30 Read block 574 len 24576 13/30 Read block 22 len 36864 14/30 Flush 15/30 Write block 66 len 28672 data: 17 16 17 17 12 11 11 16/30 Read block 568 len 32768 17/30 Read block 324 len 36864 18/30 Read block 16 len 12288 19/30 Read block 340 len 32768 20/30 Read block 481 len 28672 21/30 Read block 72 len 32768 22/30 Read block 183 len 16384 23/30 Write block 10 len 32768 data: 13 11 11 12 7 7 6 6 24/30 Write block 405 len 32768 data: 7 10 14 14 17 15 16 17 25/30 Write block 150 len 16384 data: 18 19 20 14 26/30 Read block 54 len 24576 27/30 Read block 512 len 20480 28/30 Read block 401 len 24576 29/30 Read block 389 len 32768 30/30 Write block 572 len 12288 data: 20 15 20 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 9 Done Done Done false 1008 Acked Write 2 Done Done Done false 1009 Acked Write 9 Done Done Done false 1010 Acked Write 9 Done Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 19 1 1 21 Done 4 22 22 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:00.337611611Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1904} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 174 174 174 301 301 301 F T T <--- 1 020-039 170 170 170 297 297 297 F F F 2 040-059 173 173 173 300 300 300 F F F 3 060-079 173 173 173 300 300 300 F T T <--- 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 170 170 170 297 297 297 F T T <--- 8 160-179 173 174 174 300 302 302 T F F <--- 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 172 172 172 298 298 298 F F F 12 240-259 173 173 173 300 300 300 F F F 13 260-279 173 173 173 300 300 300 F F F 14 280-299 170 170 170 297 297 297 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 173 174 174 300 302 302 T F F <--- 17 340-359 172 172 172 299 299 299 F F F 18 360-379 173 173 173 300 300 300 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 173 173 173 300 300 300 F T T <--- 21 420-439 173 173 173 300 300 300 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 173 173 173 300 300 300 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 174 174 174 301 302 302 T F F <--- 28 560-579 173 174 174 300 302 302 T T T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 174, Max flush: 302 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 170 170 170 297 297 297 F F F 2 040-059 173 173 173 300 300 300 F F F 3 060-079 175 175 175 303 303 303 F F F 4 080-099 167 167 167 292 292 292 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 175 175 175 303 303 303 F F F 8 160-179 174 174 174 302 302 302 F F F 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 172 172 172 298 298 298 F F F 12 240-259 173 173 173 300 300 300 F F F 13 260-279 173 173 173 300 300 300 F F F 14 280-299 170 170 170 297 297 297 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 174 174 174 302 302 302 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 173 173 173 300 300 300 F F F 19 380-399 169 169 169 296 296 296 F F F 20 400-419 175 175 175 303 303 303 F F F 21 420-439 173 173 173 300 300 300 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 160 160 160 279 279 279 F F F 24 480-499 173 173 173 300 300 300 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 174 174 174 302 302 302 F F F 28 560-579 175 175 175 303 303 303 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 175, Max flush: 303 Begin loop 88 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.663083715Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675105097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67542049Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675459678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675472464Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675490432Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"Crucible d17675aa-2568-4ea4-905e-78df644d5989 has session id: a3eacd74-6d3b-45ed-b0b9-4f791cf454bb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675825014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Upstairs opts: Upstairs UUID: d17675aa-2568-4ea4-905e-78df644d5989, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67585976Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Crucible stats registered with UUID: d17675aa-2568-4ea4-905e-78df644d5989","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.675875367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:04.675888953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676040172Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676067585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676082632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67621174Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676227247Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"d17675aa-2568-4ea4-905e-78df644d5989 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676249798Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676262824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67627714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676290836Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676327843Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676352204Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676367441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:46460)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676417714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33821)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676443346Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:62195)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.676459473Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"io task","client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67713387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677160662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677277916Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677522657Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677566597Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[0]R flush_numbers[0..12]: [303, 297, 300, 303, 292, 284, 291, 303, 302, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677629756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[0]R generation[0..12]: [175, 170, 173, 175, 167, 162, 166, 175, 174, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677647744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677664321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[1]R flush_numbers[0..12]: [303, 297, 300, 303, 292, 284, 291, 303, 302, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677680279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[1]R generation[0..12]: [175, 170, 173, 175, 167, 162, 166, 175, 174, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677696036Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677712183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[2]R flush_numbers[0..12]: [303, 297, 300, 303, 292, 284, 291, 303, 302, 300, 300, 298]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67772783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[2]R generation[0..12]: [175, 170, 173, 175, 167, 162, 166, 175, 174, 173, 173, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677743457Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677759555Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:04.677772841Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67779211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Next flush: 304","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677805446Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677818522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"":"downstairs","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677832808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677847665Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"0","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677861781Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"1","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677878529Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"client":"2","":"downstairs","session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677892946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677906492Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"d17675aa-2568-4ea4-905e-78df644d5989 is now active with session: a3eacd74-6d3b-45ed-b0b9-4f791cf454bb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677921388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677934014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912,"session_id":"a3eacd74-6d3b-45ed-b0b9-4f791cf454bb"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.67794747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} {"msg":"Activated sub_volume d17675aa-2568-4ea4-905e-78df644d5989","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:04.677958946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 283 len 32768 data: 10 11 13 15 16 17 20 19 02/30 Read block 221 len 28672 03/30 Write block 76 len 12288 data: 14 15 14 04/30 Read block 192 len 20480 05/30 Read block 0 len 12288 06/30 Write block 469 len 12288 data: 12 12 13 07/30 Read block 68 len 28672 08/30 Write block 379 len 24576 data: 20 20 19 22 24 21 09/30 Read block 25 len 12288 10/30 Write block 142 len 4096 data: 13 11/30 Flush 12/30 Write block 541 len 24576 data: 14 13 13 14 13 14 13/30 Write block 237 len 36864 data: 14 11 7 6 6 6 8 11 13 14/30 Write block 298 len 24576 data: 6 7 7 7 10 10 15/30 Write block 330 len 8192 data: 12 10 16/30 Read block 427 len 36864 17/30 Write block 429 len 32768 data: 15 18 19 22 21 16 18 20 18/30 Read block 485 len 16384 19/30 Write block 87 len 36864 data: 13 12 15 19 19 16 15 13 12 20/30 Read block 43 len 32768 21/30 Read block 276 len 20480 22/30 Write block 38 len 40960 data: 12 10 11 8 10 8 10 10 13 15 23/30 Write block 404 len 36864 data: 8 8 11 15 15 18 16 17 18 24/30 Flush 25/30 Write block 286 len 20480 data: 16 17 18 21 20 26/30 Read block 597 len 12288 27/30 Write block 169 len 24576 data: 17 18 19 20 19 16 28/30 Read block 535 len 24576 29/30 Read block 15 len 28672 30/30 Read block 538 len 28672 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Read 7 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:04.691521238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1912} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 176 176 170 305 305 297 F F F <--- 2 040-059 176 176 173 305 305 300 F F F <--- 3 060-079 176 176 175 304 304 303 F F F <--- 4 080-099 176 176 167 305 305 292 F F F <--- 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 176 176 175 304 304 303 F F F <--- 8 160-179 174 174 174 302 302 302 T T F <--- 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 176 176 172 305 305 298 F F F <--- 12 240-259 176 176 173 305 305 300 F F F <--- 13 260-279 173 173 173 300 300 300 F F F 14 280-299 176 176 170 305 305 297 T T F <--- 15 300-319 176 176 170 305 305 297 F F F <--- 16 320-339 176 176 174 305 305 302 F F F <--- 17 340-359 172 172 172 299 299 299 F F F 18 360-379 176 176 173 304 304 300 F F F <--- 19 380-399 176 176 169 304 304 296 F F F <--- 20 400-419 176 176 175 305 305 303 F F F <--- 21 420-439 176 176 173 305 305 300 F F F <--- 22 440-459 173 173 173 300 300 300 F F F 23 460-479 176 176 160 304 304 279 F F F <--- 24 480-499 173 173 173 300 300 300 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 176 176 174 305 305 302 F F F <--- 28 560-579 175 175 175 303 303 303 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 176, Max flush: 305 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 176 176 176 305 305 305 F F F 2 040-059 176 176 176 305 305 305 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 176 176 176 305 305 305 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 176 176 176 304 304 304 F F F 8 160-179 177 177 177 306 306 306 F F F 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 176 176 176 305 305 305 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 173 173 173 300 300 300 F F F 14 280-299 177 177 177 306 306 306 F F F 15 300-319 176 176 176 305 305 305 F F F 16 320-339 176 176 176 305 305 305 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 176 176 176 304 304 304 F F F 19 380-399 176 176 176 304 304 304 F F F 20 400-419 176 176 176 305 305 305 F F F 21 420-439 176 176 176 305 305 305 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 176 176 176 304 304 304 F F F 24 480-499 173 173 173 300 300 300 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 176 176 176 305 305 305 F F F 28 560-579 175 175 175 303 303 303 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 177, Max flush: 306 Begin loop 89 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.027438956Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:09.038078459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.038134524Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.048428119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.048722263Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.048760331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.048772626Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.048794606Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"Crucible d06d730d-5333-4ed4-a7c2-1598478f75eb has session id: 04a892af-0678-4928-b935-d11e215bc7ea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049124096Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Upstairs opts: Upstairs UUID: d06d730d-5333-4ed4-a7c2-1598478f75eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049151058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Crucible stats registered with UUID: d06d730d-5333-4ed4-a7c2-1598478f75eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049165935Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:09.049178991Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049340385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049364266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049378982Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"1","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049516635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049533132Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"d06d730d-5333-4ed4-a7c2-1598478f75eb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049553932Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049566528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049580304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"1","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.0495943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049608347Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049651616Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049674877Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"1","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:34879)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049716036Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:59553)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049750201Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"1","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:33467)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.049765518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"io task","client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.050528575Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.050591344Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"1","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.050610173Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.050928638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.050960462Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[0]R flush_numbers[0..12]: [303, 305, 305, 304, 305, 284, 291, 304, 306, 300, 300, 305]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051026882Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[0]R generation[0..12]: [175, 176, 176, 176, 176, 162, 166, 176, 177, 173, 173, 176]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.05104415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051061058Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[1]R flush_numbers[0..12]: [303, 305, 305, 304, 305, 284, 291, 304, 306, 300, 300, 305]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051076605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[1]R generation[0..12]: [175, 176, 176, 176, 176, 162, 166, 176, 177, 173, 173, 176]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051096194Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051112851Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[2]R flush_numbers[0..12]: [303, 305, 305, 304, 305, 284, 291, 304, 306, 300, 300, 305]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051128789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[2]R generation[0..12]: [175, 176, 176, 176, 176, 162, 166, 176, 177, 173, 173, 176]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051144506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051160923Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051180762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051195189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Next flush: 307","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051208075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051221321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"":"downstairs","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051235387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051247953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"0","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051262249Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"1","":"downstairs","session_id"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051279167Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"client":"2","":"downstairs","session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051293674Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.0513067Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"d06d730d-5333-4ed4-a7c2-1598478f75eb is now active with session: 04a892af-0678-4928-b935-d11e215bc7ea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051323507Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051336123Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920,"session_id":"04a892af-0678-4928-b935-d11e215bc7ea"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051349409Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} {"msg":"Activated sub_volume d06d730d-5333-4ed4-a7c2-1598478f75eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:09.051360714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 262 len 8192 data: 15 15 02/30 Read block 223 len 4096 03/30 Flush 04/30 Write block 171 len 20480 data: 20 21 20 17 15 05/30 Read block 595 len 20480 06/30 Read block 348 len 28672 07/30 Read block 344 len 8192 08/30 Flush 09/30 Write block 319 len 24576 data: 24 25 22 20 19 18 10/30 Flush 11/30 Write block 408 len 16384 data: 16 19 17 18 12/30 Read block 334 len 36864 13/30 Write block 490 len 32768 data: 17 16 15 16 19 14 15 16 14/30 Flush 15/30 Write block 364 len 36864 data: 13 13 10 11 14 16 15 13 14 16/30 Read block 51 len 36864 17/30 Write block 38 len 20480 data: 13 11 12 9 11 18/30 Read block 518 len 12288 19/30 Flush 20/30 Write block 363 len 4096 data: 11 21/30 Write block 569 len 24576 data: 21 23 21 21 16 21 22/30 Read block 329 len 24576 23/30 Read block 186 len 16384 24/30 Write block 435 len 16384 data: 19 21 20 21 25/30 Write block 222 len 20480 data: 12 14 14 16 17 26/30 Read block 125 len 24576 27/30 Read block 366 len 12288 28/30 Write block 553 len 4096 data: 20 29/30 Read block 242 len 32768 30/30 Write block 325 len 16384 data: 17 20 16 15 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 2 29 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:09.06367691Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1920} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 176 178 178 305 311 311 F F F <--- 2 040-059 176 178 178 305 311 311 F F F <--- 3 060-079 176 176 176 304 304 304 F F F 4 080-099 176 176 176 305 305 305 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 176 176 176 304 304 304 F F F 8 160-179 177 178 178 306 308 308 F F F <--- 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 176 176 176 305 305 305 F T T <--- 12 240-259 176 176 176 305 305 305 F F F 13 260-279 173 178 178 300 307 307 T F F <--- 14 280-299 177 177 177 306 306 306 F F F 15 300-319 176 178 178 305 309 309 F F F <--- 16 320-339 176 178 178 305 309 309 F T T <--- 17 340-359 172 172 172 299 299 299 F F F 18 360-379 176 178 178 304 311 311 F T T <--- 19 380-399 176 176 176 304 304 304 F F F 20 400-419 176 178 178 305 310 310 F F F <--- 21 420-439 176 176 176 305 305 305 F T T <--- 22 440-459 173 173 173 300 300 300 F F F 23 460-479 176 176 176 304 304 304 F F F 24 480-499 173 178 178 300 310 310 F F F <--- 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 176 176 176 305 305 305 F T T <--- 28 560-579 175 175 175 303 303 303 F T T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 178, Max flush: 311 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 178 178 178 311 311 311 F F F 2 040-059 178 178 178 311 311 311 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 176 176 176 305 305 305 F F F 5 100-119 162 162 162 284 284 284 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 176 176 176 304 304 304 F F F 8 160-179 178 178 178 308 308 308 F F F 9 180-199 173 173 173 300 300 300 F F F 10 200-219 173 173 173 300 300 300 F F F 11 220-239 179 179 179 312 312 312 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 177 177 177 306 306 306 F F F 15 300-319 178 178 178 309 309 309 F F F 16 320-339 179 179 179 312 312 312 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F F F 19 380-399 176 176 176 304 304 304 F F F 20 400-419 178 178 178 310 310 310 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 176 176 176 304 304 304 F F F 24 480-499 178 178 178 310 310 310 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 179, Max flush: 312 Begin loop 90 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.400870889Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:13.411536514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.411590618Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.421865055Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422172354Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422209441Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422221757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422239835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"Crucible b16b23b6-7ab9-4f48-bb01-1e92a3333b33 has session id: b2b41708-ea5f-47f3-aaff-49b8a630b625","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422594076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Upstairs opts: Upstairs UUID: b16b23b6-7ab9-4f48-bb01-1e92a3333b33, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422628122Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Crucible stats registered with UUID: b16b23b6-7ab9-4f48-bb01-1e92a3333b33","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422643659Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:13.422660257Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422808744Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422834356Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422849022Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422964375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.422980732Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"b16b23b6-7ab9-4f48-bb01-1e92a3333b33 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.4229977Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423010746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423024572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423038379Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423051935Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423066551Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423080668Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"ds_connection connected from Ok(127.0.0.1:46108)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423119886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"ds_connection connected from Ok(127.0.0.1:61482)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423136023Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"ds_connection connected from Ok(127.0.0.1:54402)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.42315099Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"io task","client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423918149Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423945862Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.423962259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424275142Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424321533Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[0]R flush_numbers[0..12]: [303, 311, 311, 304, 305, 284, 291, 304, 308, 300, 300, 312]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.42438141Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[0]R generation[0..12]: [175, 178, 178, 176, 176, 162, 166, 176, 178, 173, 173, 179]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424398648Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424415285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[1]R flush_numbers[0..12]: [303, 311, 311, 304, 305, 284, 291, 304, 308, 300, 300, 312]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424431183Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[1]R generation[0..12]: [175, 178, 178, 176, 176, 162, 166, 176, 178, 173, 173, 179]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.42444779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424464238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[2]R flush_numbers[0..12]: [303, 311, 311, 304, 305, 284, 291, 304, 308, 300, 300, 312]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424479875Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[2]R generation[0..12]: [175, 178, 178, 176, 176, 162, 166, 176, 178, 173, 173, 179]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424495342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424511439Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:13.424527747Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424548336Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Next flush: 313","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424561532Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424574528Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"":"downstairs","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424588624Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.42460145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"0","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424615607Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"1","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424629633Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"client":"2","":"downstairs","session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"All required reconciliation work is completed","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2026-03-14T05:07:13.424646451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424659907Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"b16b23b6-7ab9-4f48-bb01-1e92a3333b33 is now active with session: b2b41708-ea5f-47f3-aaff-49b8a630b625","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424674863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424687429Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928,"session_id":"b2b41708-ea5f-47f3-aaff-49b8a630b625"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424700815Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} {"msg":"Activated sub_volume b16b23b6-7ab9-4f48-bb01-1e92a3333b33","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:13.424712261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 153 len 4096 data: 15 02/30 Write block 148 len 4096 data: 18 03/30 Flush 04/30 Write block 410 len 16384 data: 18 19 19 14 05/30 Flush 06/30 Write block 300 len 4096 data: 8 07/30 Read block 306 len 4096 08/30 Write block 280 len 12288 data: 12 11 14 09/30 Read block 1 len 4096 10/30 Write block 325 len 32768 data: 18 21 17 16 13 13 11 9 11/30 Read block 440 len 36864 12/30 Read block 449 len 32768 13/30 Read block 45 len 12288 14/30 Write block 83 len 20480 data: 7 12 12 14 14 15/30 Read block 476 len 8192 16/30 Write block 462 len 16384 data: 11 13 14 13 17/30 Write block 460 len 16384 data: 10 12 12 14 18/30 Write block 195 len 32768 data: 17 16 16 13 13 13 10 10 19/30 Flush 20/30 Write block 223 len 40960 data: 15 15 17 18 17 15 16 16 19 18 21/30 Read block 407 len 16384 22/30 Read block 568 len 12288 23/30 Read block 129 len 12288 24/30 Read block 30 len 32768 25/30 Flush 26/30 Write block 91 len 40960 data: 20 17 16 14 13 12 14 11 9 10 27/30 Write block 202 len 16384 data: 11 10 10 10 28/30 Read block 55 len 28672 29/30 Write block 527 len 24576 data: 7 10 13 12 12 14 30/30 Read block 183 len 32768 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 1 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:13.436980504Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1928} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 178 178 178 311 311 311 F F F 2 040-059 178 178 178 311 311 311 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 176 180 180 305 315 315 F T T <--- 5 100-119 162 162 162 284 284 284 F T T <--- 6 120-139 166 166 166 291 291 291 F F F 7 140-159 176 180 180 304 313 313 T F F <--- 8 160-179 178 178 178 308 308 308 F F F 9 180-199 173 180 180 300 315 315 F F F <--- 10 200-219 173 180 180 300 315 315 F T T <--- 11 220-239 179 180 180 312 316 316 F F F <--- 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 177 180 180 306 315 315 F F F <--- 15 300-319 178 180 180 309 315 315 F F F <--- 16 320-339 179 180 180 312 315 315 F F F <--- 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F F F 19 380-399 176 176 176 304 304 304 F F F 20 400-419 178 180 180 310 314 314 F F F <--- 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 176 180 180 304 315 315 F F F <--- 24 480-499 178 178 178 310 310 310 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 170 170 170 297 297 297 F T T <--- 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 180, Max flush: 316 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 303 303 303 F F F 1 020-039 178 178 178 311 311 311 F F F 2 040-059 178 178 178 311 311 311 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 181 181 181 317 317 317 F F F 6 120-139 166 166 166 291 291 291 F F F 7 140-159 180 180 180 313 313 313 F F F 8 160-179 178 178 178 308 308 308 F F F 9 180-199 180 180 180 315 315 315 F F F 10 200-219 181 181 181 317 317 317 F F F 11 220-239 180 180 180 316 316 316 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 180 180 180 315 315 315 F F F 15 300-319 180 180 180 315 315 315 F F F 16 320-339 180 180 180 315 315 315 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F F F 19 380-399 176 176 176 304 304 304 F F F 20 400-419 180 180 180 314 314 314 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 180 180 180 315 315 315 F F F 24 480-499 178 178 178 310 310 310 F F F 25 500-519 172 172 172 299 299 299 F F F 26 520-539 181 181 181 317 317 317 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 181, Max flush: 317 Begin loop 91 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.772265365Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784070588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.78438165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784420627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784432753Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784450781Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"Crucible e033dc88-8731-4dc8-975f-e082d102fc48 has session id: 7514c5ec-822b-448a-8f90-2f1a59c88aa2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784815307Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Upstairs opts: Upstairs UUID: e033dc88-8731-4dc8-975f-e082d102fc48, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.784850523Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Crucible stats registered with UUID: e033dc88-8731-4dc8-975f-e082d102fc48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.78486629Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:17.784879756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785031185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785066381Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785082209Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785248584Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785264191Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"e033dc88-8731-4dc8-975f-e082d102fc48 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785288332Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785301118Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785317856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785332062Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785345678Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785359935Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785373951Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:62568)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785433488Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:54082)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785449756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:37529)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.785468044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"io task","client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786264997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.78629289Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786309987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786639627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786683998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[0]R flush_numbers[0..12]: [303, 311, 311, 304, 317, 317, 291, 313, 308, 315, 317, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786742384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[0]R generation[0..12]: [175, 178, 178, 176, 181, 181, 166, 180, 178, 180, 181, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786759682Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786779241Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[1]R flush_numbers[0..12]: [303, 311, 311, 304, 317, 317, 291, 313, 308, 315, 317, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786795318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[1]R generation[0..12]: [175, 178, 178, 176, 181, 181, 166, 180, 178, 180, 181, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786811005Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786827293Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[2]R flush_numbers[0..12]: [303, 311, 311, 304, 317, 317, 291, 313, 308, 315, 317, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.78684298Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[2]R generation[0..12]: [175, 178, 178, 176, 181, 181, 166, 180, 178, 180, 181, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786858757Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786875375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786888991Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.78690917Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Next flush: 318","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786922486Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786936032Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"":"downstairs","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786950349Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786963115Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"0","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786977361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"1","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.786993689Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"client":"2","":"downstairs","session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787011146Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787024863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"e033dc88-8731-4dc8-975f-e082d102fc48 is now active with session: 7514c5ec-822b-448a-8f90-2f1a59c88aa2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787039579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787052135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936,"session_id":"7514c5ec-822b-448a-8f90-2f1a59c88aa2"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787065831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} {"msg":"Activated sub_volume e033dc88-8731-4dc8-975f-e082d102fc48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:17.787077217Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 465 len 28672 data: 14 12 11 11 13 13 14 02/30 Read block 567 len 28672 03/30 Write block 4 len 4096 data: 11 04/30 Write block 170 len 4096 data: 19 05/30 Read block 571 len 16384 06/30 Read block 370 len 16384 07/30 Read block 95 len 32768 08/30 Read block 441 len 24576 09/30 Write block 401 len 24576 data: 11 10 7 9 9 12 10/30 Write block 304 len 4096 data: 8 11/30 Flush 12/30 Write block 504 len 36864 data: 14 16 14 14 12 13 14 12 14 13/30 Read block 297 len 40960 14/30 Write block 163 len 40960 data: 11 11 12 15 16 16 18 20 21 22 15/30 Read block 77 len 36864 16/30 Read block 356 len 4096 17/30 Read block 195 len 4096 18/30 Read block 391 len 20480 19/30 Read block 488 len 16384 20/30 Flush 21/30 Write block 13 len 40960 data: 13 8 8 7 7 4 5 4 4 6 22/30 Read block 241 len 4096 23/30 Write block 307 len 28672 data: 13 14 12 13 13 12 11 24/30 Flush 25/30 Flush 26/30 Write block 395 len 16384 data: 14 16 17 17 27/30 Write block 205 len 36864 data: 11 12 11 8 8 11 12 13 11 28/30 Read block 246 len 40960 29/30 Write block 196 len 12288 data: 17 17 14 30/30 Write block 126 len 32768 data: 18 20 20 19 20 28 24 26 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 23 1 25 Done 29 7 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:17.799699871Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1936} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 175 182 320 303 320 F T F <--- 1 020-039 182 178 182 320 311 320 F F F <--- 2 040-059 178 178 178 311 311 311 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 181 181 181 317 317 317 F F F 6 120-139 166 166 166 291 291 291 T F T <--- 7 140-159 180 180 180 313 313 313 F F F 8 160-179 182 178 182 319 308 319 F T F <--- 9 180-199 180 180 180 315 315 315 T F T <--- 10 200-219 181 181 181 317 317 317 T F T <--- 11 220-239 180 180 180 316 316 316 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 180 180 180 315 315 315 F F F 15 300-319 182 180 182 320 315 320 F F F <--- 16 320-339 180 180 180 315 315 315 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F F F 19 380-399 176 176 176 304 304 304 T F T <--- 20 400-419 182 180 182 318 314 318 F F F <--- 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 182 180 182 318 315 318 F T F <--- 24 480-499 178 178 178 310 310 310 F F F 25 500-519 182 172 182 319 299 319 F F F <--- 26 520-539 181 181 181 317 317 317 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 182, Max flush: 320 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 182 182 182 320 320 320 F F F 2 040-059 178 178 178 311 311 311 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 181 181 181 317 317 317 F F F 6 120-139 183 183 183 321 321 321 F F F 7 140-159 180 180 180 313 313 313 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 183 183 183 321 321 321 F F F 10 200-219 183 183 183 321 321 321 F F F 11 220-239 180 180 180 316 316 316 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 180 180 180 315 315 315 F F F 15 300-319 182 182 182 320 320 320 F F F 16 320-339 180 180 180 315 315 315 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F F F 19 380-399 183 183 183 321 321 321 F F F 20 400-419 182 182 182 318 318 318 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 173 173 300 300 300 F F F 23 460-479 182 182 182 318 318 318 F F F 24 480-499 178 178 178 310 310 310 F F F 25 500-519 182 182 182 319 319 319 F F F 26 520-539 181 181 181 317 317 317 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 183, Max flush: 321 Begin loop 92 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.123175828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:22.133870135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.1339246Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.144407151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.144703946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.144745165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.144757701Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.144775609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"Crucible fd39e20f-2555-4fed-a1e1-63e018c0bf4b has session id: 4a337d44-f927-47e4-9562-4d1c564e6c08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145116784Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Upstairs opts: Upstairs UUID: fd39e20f-2555-4fed-a1e1-63e018c0bf4b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145153941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Crucible stats registered with UUID: fd39e20f-2555-4fed-a1e1-63e018c0bf4b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145169978Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:22.145183284Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145334393Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145357764Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.14537261Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145506151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145521448Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"fd39e20f-2555-4fed-a1e1-63e018c0bf4b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145539196Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145551782Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145565868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145580135Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145593731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145607868Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145622164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"ds_connection connected from Ok(127.0.0.1:51756)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145650397Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"ds_connection connected from Ok(127.0.0.1:41966)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.145665214Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"ds_connection connected from Ok(127.0.0.1:56775)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.14567997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"io task","client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146459095Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146510558Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146529007Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146834676Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146881707Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[0]R flush_numbers[0..12]: [320, 320, 311, 304, 317, 317, 321, 313, 319, 321, 321, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146927408Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[0]R generation[0..12]: [182, 182, 178, 176, 181, 181, 183, 180, 182, 183, 183, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146944536Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146961414Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[1]R flush_numbers[0..12]: [320, 320, 311, 304, 317, 317, 321, 313, 319, 321, 321, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146977481Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[1]R generation[0..12]: [182, 182, 178, 176, 181, 181, 183, 180, 182, 183, 183, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.146993198Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147009165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[2]R flush_numbers[0..12]: [320, 320, 311, 304, 317, 317, 321, 313, 319, 321, 321, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147027854Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[2]R generation[0..12]: [182, 182, 178, 176, 181, 181, 183, 180, 182, 183, 183, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147043491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147059738Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Max found gen is 184","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-03-14T05:07:22.147079517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147093814Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Next flush: 322","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.14710704Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147120216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"":"downstairs","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147134502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147147288Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"0","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147161715Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"1","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:07:22.147176902Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"client":"2","":"downstairs","session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.14719448Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147207766Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"fd39e20f-2555-4fed-a1e1-63e018c0bf4b is now active with session: 4a337d44-f927-47e4-9562-4d1c564e6c08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147222632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147236019Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944,"session_id":"4a337d44-f927-47e4-9562-4d1c564e6c08"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147249945Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} {"msg":"Activated sub_volume fd39e20f-2555-4fed-a1e1-63e018c0bf4b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:22.147263001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 481 len 36864 data: 18 16 13 9 10 9 8 12 16 02/30 Read block 285 len 4096 03/30 Write block 454 len 36864 data: 11 8 8 10 11 10 11 13 13 04/30 Read block 24 len 20480 05/30 Write block 38 len 20480 data: 14 12 13 10 12 06/30 Read block 9 len 20480 07/30 Read block 71 len 32768 08/30 Flush 09/30 Write block 328 len 4096 data: 17 10/30 Read block 244 len 8192 11/30 Write block 100 len 16384 data: 11 10 9 9 12/30 Read block 555 len 20480 13/30 Read block 189 len 28672 14/30 Read block 21 len 12288 15/30 Write block 201 len 24576 data: 11 12 11 11 12 13 16/30 Read block 189 len 40960 17/30 Read block 39 len 16384 18/30 Read block 497 len 20480 19/30 Read block 472 len 4096 20/30 Read block 529 len 24576 21/30 Read block 187 len 4096 22/30 Read block 468 len 32768 23/30 Write block 112 len 20480 data: 13 12 10 11 12 24/30 Write block 367 len 12288 data: 12 15 17 25/30 Read block 278 len 32768 26/30 Read block 254 len 40960 27/30 Write block 404 len 16384 data: 10 10 13 16 28/30 Read block 505 len 24576 29/30 Write block 329 len 4096 data: 14 30/30 Read block 403 len 36864 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Sent Done Done false 1001 Acked Read 1 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 1 Sent Done Done false 1029 Acked Read 9 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1007)) Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:22.159644946Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1944} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 182 184 184 320 322 322 F F F <--- 2 040-059 178 184 184 311 322 322 F F F <--- 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 181 181 181 317 317 317 F T T <--- 6 120-139 183 183 183 321 321 321 F F F 7 140-159 180 180 180 313 313 313 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 183 183 183 321 321 321 F F F 10 200-219 183 183 183 321 321 321 F T T <--- 11 220-239 180 180 180 316 316 316 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 180 180 180 315 315 315 F F F 15 300-319 182 182 182 320 320 320 F F F 16 320-339 180 180 180 315 315 315 F T T <--- 17 340-359 172 172 172 299 299 299 F F F 18 360-379 179 179 179 312 312 312 F T T <--- 19 380-399 183 183 183 321 321 321 F F F 20 400-419 182 182 182 318 318 318 F T T <--- 21 420-439 179 179 179 312 312 312 F F F 22 440-459 173 184 184 300 322 322 F F F <--- 23 460-479 182 184 184 318 322 322 F F F <--- 24 480-499 178 184 184 310 322 322 F F F <--- 25 500-519 182 182 182 319 319 319 F F F 26 520-539 181 181 181 317 317 317 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 184, Max flush: 322 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F F F 2 040-059 184 184 184 322 322 322 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 185 185 185 323 323 323 F F F 6 120-139 183 183 183 321 321 321 F F F 7 140-159 180 180 180 313 313 313 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 183 183 183 321 321 321 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 180 180 180 316 316 316 F F F 12 240-259 176 176 176 305 305 305 F F F 13 260-279 178 178 178 307 307 307 F F F 14 280-299 180 180 180 315 315 315 F F F 15 300-319 182 182 182 320 320 320 F F F 16 320-339 185 185 185 323 323 323 F F F 17 340-359 172 172 172 299 299 299 F F F 18 360-379 185 185 185 323 323 323 F F F 19 380-399 183 183 183 321 321 321 F F F 20 400-419 185 185 185 323 323 323 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 184 184 184 322 322 322 F F F 23 460-479 184 184 184 322 322 322 F F F 24 480-499 184 184 184 322 322 322 F F F 25 500-519 182 182 182 319 319 319 F F F 26 520-539 181 181 181 317 317 317 F F F 27 540-559 179 179 179 312 312 312 F F F 28 560-579 179 179 179 312 312 312 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 185, Max flush: 323 Begin loop 93 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.492247627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504103423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504430892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.50447015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504482796Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504500664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"Crucible 7c011d03-6c1f-437c-aebe-3d130d8da3eb has session id: 26830f9e-e771-47da-8c5d-afc464242f39","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504837857Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Upstairs opts: Upstairs UUID: 7c011d03-6c1f-437c-aebe-3d130d8da3eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.50486693Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Crucible stats registered with UUID: 7c011d03-6c1f-437c-aebe-3d130d8da3eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.504881697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:26.504894663Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505070263Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505094914Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505109731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505238169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505252816Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"7c011d03-6c1f-437c-aebe-3d130d8da3eb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.50528367Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505297806Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505312253Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505326069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505339886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505354092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505368459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"ds_connection connected from Ok(127.0.0.1:35390)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505395311Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"ds_connection connected from Ok(127.0.0.1:56938)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505410388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"ds_connection connected from Ok(127.0.0.1:58235)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.505425395Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"io task","client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506224328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506249109Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506266187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506597298Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506621789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[0]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 321, 313, 319, 321, 323, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.50664502Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[0]R generation[0..12]: [182, 184, 184, 176, 181, 185, 183, 180, 182, 183, 185, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506660997Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506677614Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[1]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 321, 313, 319, 321, 323, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506693572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[1]R generation[0..12]: [182, 184, 184, 176, 181, 185, 183, 180, 182, 183, 185, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506709259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506725746Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[2]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 321, 313, 319, 321, 323, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506741554Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[2]R generation[0..12]: [182, 184, 184, 176, 181, 185, 183, 180, 182, 183, 185, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506757121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506773348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506786724Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:26.50680065Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Next flush: 324","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506820159Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506833185Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"":"downstairs","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506848762Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506861308Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"0","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506875314Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"1","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506889031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"client":"2","":"downstairs","session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506903107Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:07:26.506916303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"7c011d03-6c1f-437c-aebe-3d130d8da3eb is now active with session: 26830f9e-e771-47da-8c5d-afc464242f39","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.50693218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506945136Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952,"session_id":"26830f9e-e771-47da-8c5d-afc464242f39"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506958722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} {"msg":"Activated sub_volume 7c011d03-6c1f-437c-aebe-3d130d8da3eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:26.506970218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 253 len 36864 data: 20 22 20 18 16 15 13 14 14 02/30 Read block 148 len 24576 03/30 Write block 535 len 32768 data: 15 11 10 13 16 15 15 14 04/30 Read block 184 len 36864 05/30 Read block 557 len 4096 06/30 Flush 07/30 Write block 356 len 32768 data: 15 15 18 16 15 11 11 12 08/30 Read block 428 len 24576 09/30 Read block 66 len 24576 10/30 Read block 458 len 12288 11/30 Write block 243 len 20480 data: 9 12 14 16 16 12/30 Read block 521 len 8192 13/30 Write block 315 len 4096 data: 15 14/30 Read block 154 len 32768 15/30 Write block 271 len 36864 data: 15 16 16 16 16 18 16 14 14 16/30 Write block 457 len 36864 data: 11 12 11 12 14 14 15 15 15 17/30 Write block 293 len 4096 data: 16 18/30 Read block 23 len 36864 19/30 Flush 20/30 Write block 545 len 24576 data: 14 15 15 16 18 22 21/30 Write block 565 len 12288 data: 14 16 15 22/30 Write block 239 len 12288 data: 8 7 7 23/30 Flush 24/30 Write block 137 len 32768 data: 13 15 14 15 11 14 14 14 25/30 Write block 473 len 8192 data: 12 10 26/30 Write block 498 len 8192 data: 19 16 27/30 Read block 456 len 36864 28/30 Read block 258 len 28672 29/30 Read block 282 len 40960 30/30 Read block 578 len 12288 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 2 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:26.519821487Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1952} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F F F 2 040-059 184 184 184 322 322 322 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 185 185 185 323 323 323 F F F 6 120-139 183 183 183 321 321 321 T F T <--- 7 140-159 180 180 180 313 313 313 T F T <--- 8 160-179 182 182 182 319 319 319 F F F 9 180-199 183 183 183 321 321 321 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 180 186 326 316 326 F F F <--- 12 240-259 186 176 186 326 305 326 F T F <--- 13 260-279 186 178 186 325 307 325 F T F <--- 14 280-299 186 180 186 325 315 325 F F F <--- 15 300-319 186 182 186 325 320 325 F F F <--- 16 320-339 185 185 185 323 323 323 F F F 17 340-359 186 172 186 325 299 325 F F F <--- 18 360-379 186 185 186 325 323 325 F F F <--- 19 380-399 183 183 183 321 321 321 F F F 20 400-419 185 185 185 323 323 323 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 184 186 325 322 325 F F F <--- 23 460-479 186 184 186 325 322 325 T F T <--- 24 480-499 184 184 184 322 322 322 T F T <--- 25 500-519 182 182 182 319 319 319 F F F 26 520-539 186 181 186 324 317 324 F T F <--- 27 540-559 186 179 186 326 312 326 F T F <--- 28 560-579 186 179 186 326 312 326 F F F <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 186, Max flush: 326 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F F F 2 040-059 184 184 184 322 322 322 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 181 181 181 317 317 317 F F F 5 100-119 185 185 185 323 323 323 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 183 183 183 321 321 321 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 186 186 326 326 326 F F F 12 240-259 186 186 186 326 326 326 F F F 13 260-279 186 186 186 325 325 325 F F F 14 280-299 186 186 186 325 325 325 F F F 15 300-319 186 186 186 325 325 325 F F F 16 320-339 185 185 185 323 323 323 F F F 17 340-359 186 186 186 325 325 325 F F F 18 360-379 186 186 186 325 325 325 F F F 19 380-399 183 183 183 321 321 321 F F F 20 400-419 185 185 185 323 323 323 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 187 187 187 327 327 327 F F F 24 480-499 187 187 187 327 327 327 F F F 25 500-519 182 182 182 319 319 319 F F F 26 520-539 186 186 186 324 324 324 F F F 27 540-559 186 186 186 326 326 326 F F F 28 560-579 186 186 186 326 326 326 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 187, Max flush: 327 Begin loop 94 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.853997873Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.865662752Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.86598792Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866029789Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866043175Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866061273Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"Crucible 60b3e459-1b93-42e3-867a-16d58708eb87 has session id: 4ebe6d22-d103-45ef-8fde-2a73c622585c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866433252Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Upstairs opts: Upstairs UUID: 60b3e459-1b93-42e3-867a-16d58708eb87, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866467828Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Crucible stats registered with UUID: 60b3e459-1b93-42e3-867a-16d58708eb87","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866483165Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:30.866496111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866650921Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866674892Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866689539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866811054Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866826451Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"60b3e459-1b93-42e3-867a-16d58708eb87 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866844079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866856825Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866871392Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866885588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866899525Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866913721Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866930549Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38320)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866968056Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"ds_connection connected from Ok(127.0.0.1:52464)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.866984113Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46248)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.867094463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"io task","client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.867765239Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.867792341Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.867808939Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868153175Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868203788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[0]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 327, 327, 319, 321, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868242516Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[0]R generation[0..12]: [182, 184, 184, 176, 181, 185, 187, 187, 182, 183, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868259824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868276561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[1]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 327, 327, 319, 321, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868292339Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[1]R generation[0..12]: [182, 184, 184, 176, 181, 185, 187, 187, 182, 183, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868307856Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868326604Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[2]R flush_numbers[0..12]: [320, 322, 322, 304, 317, 323, 327, 327, 319, 321, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868342782Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[2]R generation[0..12]: [182, 184, 184, 176, 181, 185, 187, 187, 182, 183, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868358649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868375226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_idWait for a query_work_queue command to finish before sending IO ":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868395756Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868409782Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Next flush: 328","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868423478Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868437295Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"":"downstairs","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868451331Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868464187Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"0","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868477953Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"1","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868491639Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"client":"2","":"downstairs","session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868508697Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868522043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"60b3e459-1b93-42e3-867a-16d58708eb87 is now active with session: 4ebe6d22-d103-45ef-8fde-2a73c622585c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.86853719Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868551867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960,"session_id":"4ebe6d22-d103-45ef-8fde-2a73c622585c"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868572546Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} {"msg":"Activated sub_volume 60b3e459-1b93-42e3-867a-16d58708eb87","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:30.868584111Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 385 len 40960 data: 18 17 18 16 16 16 16 17 18 18 02/30 Read block 579 len 40960 03/30 Write block 182 len 24576 data: 13 15 14 16 18 19 04/30 Write block 296 len 28672 data: 8 7 7 8 9 8 11 05/30 Read block 52 len 36864 06/30 Write block 195 len 8192 data: 18 18 07/30 Read block 282 len 8192 08/30 Read block 445 len 16384 09/30 Write block 88 len 16384 data: 13 16 20 21 10/30 Read block 530 len 12288 11/30 Read block 91 len 16384 12/30 Flush 13/30 Write block 389 len 4096 data: 17 14/30 Read block 331 len 16384 15/30 Write block 286 len 36864 data: 17 18 19 22 21 18 17 17 14 16/30 Write block 409 len 36864 data: 20 19 20 20 15 10 11 9 10 17/30 Write block 317 len 32768 data: 18 22 25 26 23 21 20 19 18/30 Read block 391 len 40960 19/30 Read block 518 len 24576 20/30 Write block 44 len 40960 data: 11 11 14 16 17 15 13 16 13 16 21/30 Write block 460 len 20480 data: 13 15 15 16 16 22/30 Read block 211 len 4096 23/30 Read block 458 len 4096 24/30 Flush 25/30 Write block 376 len 20480 data: 12 16 17 21 21 26/30 Read block 468 len 24576 27/30 Write block 520 len 28672 data: 9 8 11 10 9 11 8 28/30 Write block 330 len 28672 data: 14 12 10 10 8 10 12 29/30 Read block 518 len 28672 30/30 Read block 269 len 36864 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 6 Done Sent Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 30 2 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:30.881463683Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1960} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F F F 2 040-059 188 184 188 329 322 329 F F F <--- 3 060-079 176 176 176 304 304 304 F F F 4 080-099 188 181 188 328 317 328 F F F <--- 5 100-119 185 185 185 323 323 323 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 188 183 188 328 321 328 F F F <--- 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 186 186 326 326 326 F F F 12 240-259 186 186 186 326 326 326 F F F 13 260-279 186 186 186 325 325 325 F F F 14 280-299 188 186 188 329 325 329 F F F <--- 15 300-319 188 186 188 329 325 329 F F F <--- 16 320-339 188 185 188 329 323 329 T F T <--- 17 340-359 186 186 186 325 325 325 F F F 18 360-379 186 186 186 325 325 325 T F T <--- 19 380-399 188 183 188 329 321 329 T T T <--- 20 400-419 188 185 188 329 323 329 F F F <--- 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 188 187 188 329 327 329 F F F <--- 24 480-499 187 187 187 327 327 327 F F F 25 500-519 182 182 182 319 319 319 F F F 26 520-539 186 186 186 324 324 324 T F T <--- 27 540-559 186 186 186 326 326 326 F F F 28 560-579 186 186 186 326 326 326 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 188, Max flush: 329 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F F F 2 040-059 188 188 188 329 329 329 F F F 3 060-079 176 176 176 304 304 304 F F F 4 080-099 188 188 188 328 328 328 F F F 5 100-119 185 185 185 323 323 323 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 188 188 188 328 328 328 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 186 186 326 326 326 F F F 12 240-259 186 186 186 326 326 326 F F F 13 260-279 186 186 186 325 325 325 F F F 14 280-299 188 188 188 329 329 329 F F F 15 300-319 188 188 188 329 329 329 F F F 16 320-339 189 189 189 330 330 330 F F F 17 340-359 186 186 186 325 325 325 F F F 18 360-379 189 189 189 330 330 330 F F F 19 380-399 189 189 189 330 330 330 F F F 20 400-419 188 188 188 329 329 329 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 188 188 188 329 329 329 F F F 24 480-499 187 187 187 327 327 327 F F F 25 500-519 182 182 182 319 319 319 F F F 26 520-539 189 189 189 330 330 330 F F F 27 540-559 186 186 186 326 326 326 F F F 28 560-579 186 186 186 326 326 326 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 189, Max flush: 330 Begin loop 95 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.205004808Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:35.21564385Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.215691082Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22607834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226373304Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226410831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226422786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226440514Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"Crucible 7aea3218-802c-4647-b138-fc6d94f55fb6 has session id: 474fc544-7572-49d4-8255-769f6d156d12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22682586Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Upstairs opts: Upstairs UUID: 7aea3218-802c-4647-b138-fc6d94f55fb6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226860596Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Crucible stats registered with UUID: 7aea3218-802c-4647-b138-fc6d94f55fb6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.226876783Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:35.226890419Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227044119Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227080366Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227095993Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227196539Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227211075Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"7aea3218-802c-4647-b138-fc6d94f55fb6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227232755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227245561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227259927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227273834Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22728765Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227326318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227349538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"ds_connection connected from Ok(127.0.0.1:41212)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227365315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"ds_connection connected from Ok(127.0.0.1:36277)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.227516574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"ds_connection connected from Ok(127.0.0.1:60432)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22755121Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"io task","client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228070256Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228103681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228120579Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22847476Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228494689Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[0]R flush_numbers[0..12]: [320, 322, 329, 304, 328, 323, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228519181Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[0]R generation[0..12]: [182, 184, 188, 176, 188, 185, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228535328Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228551545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[1]R flush_numbers[0..12]: [320, 322, 329, 304, 328, 323, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228567473Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[1]R generation[0..12]: [182, 184, 188, 176, 188, 185, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22858318Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228599487Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[2]R flush_numbers[0..12]: [320, 322, 329, 304, 328, 323, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228615104Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[2]R generation[0..12]: [182, 184, 188, 176, 188, 185, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228630941Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228647259Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228663656Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228683625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228697552Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228710998Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"":"downstairs","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228725134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.22873776Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"0","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228751826Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"1","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228765453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"client":"2","":"downstairs","session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228779839Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228796847Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"7aea3218-802c-4647-b138-fc6d94f55fb6 is now active with session: 474fc544-7572-49d4-8255-769f6d156d12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228812014Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228824459Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968,"session_id":"474fc544-7572-49d4-8255-769f6d156d12"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228837786Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} {"msg":"Activated sub_volume 7aea3218-802c-4647-b138-fc6d94f55fb6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:35.228849401Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 463 len 28672 data: 17 17 16 13 12 12 14 02/30 Write block 565 len 24576 data: 15 17 16 17 22 24 03/30 Write block 55 len 24576 data: 13 11 10 11 11 11 04/30 Read block 142 len 8192 05/30 Write block 302 len 32768 data: 12 11 9 10 11 14 15 13 06/30 Read block 19 len 36864 07/30 Write block 270 len 16384 data: 14 16 17 17 08/30 Flush 09/30 Write block 110 len 28672 data: 13 13 14 13 11 12 13 10/30 Flush 11/30 Write block 511 len 32768 data: 13 15 13 16 12 13 12 9 12/30 Write block 42 len 40960 data: 13 9 12 12 15 17 18 16 14 17 13/30 Read block 467 len 12288 14/30 Read block 237 len 36864 15/30 Write block 543 len 24576 data: 14 15 15 16 16 17 16/30 Read block 93 len 28672 17/30 Write block 554 len 40960 data: 18 17 14 12 11 9 8 10 11 10 18/30 Write block 248 len 28672 data: 17 19 17 15 20 21 23 19/30 Write block 377 len 12288 data: 17 18 22 20/30 Read block 94 len 20480 21/30 Read block 39 len 32768 22/30 Write block 318 len 20480 data: 23 26 27 24 22 23/30 Read block 427 len 20480 24/30 Read block 81 len 8192 25/30 Read block 176 len 28672 26/30 Write block 379 len 24576 data: 23 22 20 23 25 22 27/30 Read block 251 len 16384 28/30 Read block 595 len 16384 29/30 Read block 427 len 32768 30/30 Write block 31 len 36864 data: 12 10 11 14 12 11 11 15 13 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Write 6 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1009)) Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:35.242372876Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1968} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 184 184 184 322 322 322 F T T <--- 2 040-059 188 190 190 329 331 331 F T T <--- 3 060-079 176 190 190 304 331 331 F F F <--- 4 080-099 188 188 188 328 328 328 F F F 5 100-119 185 190 190 323 332 332 F F F <--- 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 188 188 188 328 328 328 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 186 186 326 326 326 F F F 12 240-259 186 186 186 326 326 326 F T T <--- 13 260-279 186 190 190 325 331 331 F F F <--- 14 280-299 188 188 188 329 329 329 F F F 15 300-319 188 190 190 329 331 331 F T T <--- 16 320-339 189 189 189 330 330 330 F T T <--- 17 340-359 186 186 186 325 325 325 F F F 18 360-379 189 189 189 330 330 330 F T T <--- 19 380-399 189 189 189 330 330 330 F T T <--- 20 400-419 188 188 188 329 329 329 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 188 190 190 329 331 331 F F F <--- 24 480-499 187 187 187 327 327 327 F F F 25 500-519 182 182 182 319 319 319 F T T <--- 26 520-539 189 189 189 330 330 330 F F F 27 540-559 186 186 186 326 326 326 F T T <--- 28 560-579 186 190 190 326 331 331 F T T <--- 29 580-599 147 147 147 254 254 254 F F F Max gen: 190, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 191 191 191 333 333 333 F F F 3 060-079 190 190 190 331 331 331 F F F 4 080-099 188 188 188 328 328 328 F F F 5 100-119 190 190 190 332 332 332 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 182 182 182 319 319 319 F F F 9 180-199 188 188 188 328 328 328 F F F 10 200-219 185 185 185 323 323 323 F F F 11 220-239 186 186 186 326 326 326 F F F 12 240-259 191 191 191 333 333 333 F F F 13 260-279 190 190 190 331 331 331 F F F 14 280-299 188 188 188 329 329 329 F F F 15 300-319 191 191 191 333 333 333 F F F 16 320-339 191 191 191 333 333 333 F F F 17 340-359 186 186 186 325 325 325 F F F 18 360-379 191 191 191 333 333 333 F F F 19 380-399 191 191 191 333 333 333 F F F 20 400-419 188 188 188 329 329 329 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 190 190 190 331 331 331 F F F 24 480-499 187 187 187 327 327 327 F F F 25 500-519 191 191 191 333 333 333 F F F 26 520-539 189 189 189 330 330 330 F F F 27 540-559 191 191 191 333 333 333 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 191, Max flush: 333 Begin loop 96 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.586407649Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.598354927Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.59871279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.598752588Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.598764863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.598782361Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"Crucible b3eb7fd3-0b54-41da-962a-00145302505d has session id: 2ad622f4-a9e6-4e7f-8928-6910b1b71c44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.59913189Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Upstairs opts: Upstairs UUID: b3eb7fd3-0b54-41da-962a-00145302505d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599157722Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Crucible stats registered with UUID: b3eb7fd3-0b54-41da-962a-00145302505d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599172069Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:39.599185285Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599361705Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599396631Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599412538Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.59950466Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599518896Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"b3eb7fd3-0b54-41da-962a-00145302505d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599536635Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.5995493Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599566218Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599580394Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.59959369Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599607827Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599621943Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:34513)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599685522Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:64116)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.59970166Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:43644)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.599728222Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"io task","client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600498112Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600529226Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600569134Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600857266Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600898424Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[0]R flush_numbers[0..12]: [320, 333, 333, 331, 328, 332, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600963314Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[0]R generation[0..12]: [182, 191, 191, 190, 188, 190, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.600980632Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601010886Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[1]R flush_numbers[0..12]: [320, 333, 333, 331, 328, 332, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601027043Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[1]R generation[0..12]: [182, 191, 191, 190, 188, 190, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.60104269Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601058987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[2]R flush_numbers[0..12]: [320, 333, 333, 331, 328, 332, 327, 327, 319, 328, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601074675Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[2]R generation[0..12]: [182, 191, 191, 190, 188, 190, 187, 187, 182, 188, 185, 186]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601090642Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601106769Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601120415Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:39.601134582Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.6011518Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601165416Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"":"downstairs","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601179342Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601192028Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"0","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601206044Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"1","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601221731Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"client":"2","":"downstairs","session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601236048Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601250505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"b3eb7fd3-0b54-41da-962a-00145302505d is now active with session: 2ad622f4-a9e6-4e7f-8928-6910b1b71c44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601265842Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601278417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976,"session_id":"2ad622f4-a9e6-4e7f-8928-6910b1b71c44"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601291793Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} {"msg":"Activated sub_volume b3eb7fd3-0b54-41da-962a-00145302505d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:39.601303199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 341 len 4096 data: 17 02/30 Read block 2 len 24576 03/30 Write block 339 len 12288 data: 16 17 18 04/30 Write block 351 len 28672 data: 15 14 14 15 15 16 16 05/30 Write block 224 len 12288 data: 16 18 19 06/30 Write block 171 len 32768 data: 22 23 21 18 16 14 15 14 07/30 Read block 17 len 28672 08/30 Read block 366 len 28672 09/30 Read block 260 len 4096 10/30 Read block 84 len 28672 11/30 Flush 12/30 Write block 355 len 12288 data: 16 17 17 13/30 Write block 493 len 28672 data: 17 20 15 16 17 20 17 14/30 Read block 490 len 20480 15/30 Write block 54 len 4096 data: 15 16/30 Write block 506 len 32768 data: 15 15 13 14 15 14 16 14 17/30 Write block 293 len 36864 data: 18 15 8 9 8 8 9 10 9 18/30 Write block 83 len 20480 data: 8 13 13 15 15 19/30 Read block 198 len 40960 20/30 Read block 349 len 16384 21/30 Read block 562 len 20480 22/30 Write block 330 len 28672 data: 15 13 11 11 9 11 13 23/30 Write block 325 len 40960 data: 19 22 18 18 15 16 14 12 12 10 24/30 Write block 360 len 8192 data: 16 12 25/30 Write block 207 len 24576 data: 12 9 9 12 13 14 26/30 Flush 27/30 Write block 361 len 24576 data: 13 12 13 14 14 11 28/30 Write block 11 len 16384 data: 12 12 14 9 29/30 Read block 69 len 16384 30/30 Read block 494 len 12288 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 3 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Write 8 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:39.612536771Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1976} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 320 320 320 T T F <--- 1 020-039 191 191 191 333 333 333 F F F 2 040-059 192 192 191 335 335 333 F F F <--- 3 060-079 190 190 190 331 331 331 F F F 4 080-099 192 192 188 335 335 328 F F F <--- 5 100-119 190 190 190 332 332 332 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 192 192 182 334 334 319 F F F <--- 9 180-199 188 188 188 328 328 328 F F F 10 200-219 192 192 185 335 335 323 F F F <--- 11 220-239 192 192 186 334 334 326 F F F <--- 12 240-259 191 191 191 333 333 333 F F F 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 188 335 335 329 F F F <--- 15 300-319 192 192 191 335 335 333 F F F <--- 16 320-339 192 192 191 335 335 333 F F F <--- 17 340-359 192 192 186 335 335 325 F F T <--- 18 360-379 192 192 191 335 335 333 T T F <--- 19 380-399 191 191 191 333 333 333 F F F 20 400-419 188 188 188 329 329 329 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 190 190 190 331 331 331 F F F 24 480-499 192 192 187 335 335 327 F F F <--- 25 500-519 192 192 191 335 335 333 F F F <--- 26 520-539 189 189 189 330 330 330 F F F 27 540-559 191 191 191 333 333 333 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 192, Max flush: 335 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 192 192 192 335 335 335 F F F 3 060-079 190 190 190 331 331 331 F F F 4 080-099 192 192 192 335 335 335 F F F 5 100-119 190 190 190 332 332 332 F F F 6 120-139 187 187 187 327 327 327 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 192 192 192 334 334 334 F F F 9 180-199 188 188 188 328 328 328 F F F 10 200-219 192 192 192 335 335 335 F F F 11 220-239 192 192 192 334 334 334 F F F 12 240-259 191 191 191 333 333 333 F F F 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 192 335 335 335 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 192 192 192 335 335 335 F F F 18 360-379 193 193 193 336 336 336 F F F 19 380-399 191 191 191 333 333 333 F F F 20 400-419 188 188 188 329 329 329 F F F 21 420-439 179 179 179 312 312 312 F F F 22 440-459 186 186 186 325 325 325 F F F 23 460-479 190 190 190 331 331 331 F F F 24 480-499 192 192 192 335 335 335 F F F 25 500-519 192 192 192 335 335 335 F F F 26 520-539 189 189 189 330 330 330 F F F 27 540-559 191 191 191 333 333 333 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 193, Max flush: 336 Begin loop 97 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.938160384Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.949790388Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950115526Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950157925Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950171031Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950189199Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"Crucible 36774e2f-0b99-4422-af9c-dec114b0ef2d has session id: 0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950559638Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Upstairs opts: Upstairs UUID: 36774e2f-0b99-4422-af9c-dec114b0ef2d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950590322Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Crucible stats registered with UUID: 36774e2f-0b99-4422-af9c-dec114b0ef2d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.95060844Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:43.950622126Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950768423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"2","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950803409Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950818916Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950908867Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950923994Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"36774e2f-0b99-4422-af9c-dec114b0ef2d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950943542Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950956568Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950970755Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950984741Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"2","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.950998547Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"2","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951013164Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951027421Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:35868)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951042417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:63635)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951057484Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:39084)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951075422Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"io task","client":"2","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951709821Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951739315Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.951802303Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"2","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952083831Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952097898Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[0]R flush_numbers[0..12]: [336, 333, 335, 331, 335, 332, 327, 327, 334, 328, 335, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952185248Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[0]R generation[0..12]: [193, 191, 192, 190, 192, 190, 187, 187, 192, 188, 192, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952202375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952219073Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[1]R flush_numbers[0..12]: [336, 333, 335, 331, 335, 332, 327, 327, 334, 328, 335, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.95223512Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[1]R generation[0..12]: [193, 191, 192, 190, 192, 190, 187, 187, 192, 188, 192, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952251068Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952267155Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[2]R flush_numbers[0..12]: [336, 333, 335, 331, 335, 332, 327, 327, 334, 328, 335, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952282992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[2]R generation[0..12]: [193, 191, 192, 190, 192, 190, 187, 187, 192, 188, 192, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952302101Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952318788Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:43.952332505Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952352474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Next flush: 337","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.95236583Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952379216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"":"downstairs","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952393572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952405768Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"0","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952420375Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"1","":"downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952434571Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"client":"2","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952451609Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952465145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"36774e2f-0b99-4422-af9c-dec114b0ef2d is now active with session: 0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952480192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952492908Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984,"session_id":"0f86c409-6dc1-4b5f-80af-6cfb4dd4c4e6"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952506274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} {"msg":"Activated sub_volume 36774e2f-0b99-4422-af9c-dec114b0ef2d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:43.952517879Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 435 len 4096 data: 20 02/30 Write block 175 len 4096 data: 17 03/30 Write block 47 len 12288 data: 18 19 17 04/30 Flush 05/30 Write block 537 len 4096 data: 11 06/30 Write block 438 len 16384 data: 22 19 17 11 07/30 Write block 413 len 36864 data: 16 11 12 10 11 8 9 12 12 08/30 Write block 541 len 24576 data: 16 15 15 16 16 17 09/30 Write block 115 len 40960 data: 13 14 12 12 14 14 15 15 15 18 10/30 Read block 354 len 16384 11/30 Read block 579 len 8192 12/30 Write block 401 len 36864 data: 12 11 8 11 11 14 17 17 21 13/30 Read block 204 len 8192 14/30 Write block 254 len 8192 data: 24 21 15/30 Read block 75 len 8192 16/30 Write block 595 len 12288 data: 10 10 9 17/30 Read block 576 len 4096 18/30 Flush 19/30 Write block 227 len 32768 data: 18 16 17 17 20 19 16 16 20/30 Read block 253 len 4096 21/30 Write block 198 len 12288 data: 15 14 14 22/30 Read block 316 len 20480 23/30 Write block 399 len 36864 data: 16 13 13 12 9 12 12 15 18 24/30 Read block 388 len 36864 25/30 Read block 588 len 32768 26/30 Read block 179 len 36864 27/30 Write block 232 len 24576 data: 20 17 17 15 15 15 28/30 Read block 451 len 16384 29/30 Write block 460 len 28672 data: 14 16 16 18 18 17 14 30/30 Write block 461 len 12288 data: 17 17 19 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 1 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:43.964179216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1984} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 194 194 192 337 337 335 F F F <--- 3 060-079 190 190 190 331 331 331 F F F 4 080-099 192 192 192 335 335 335 F F F 5 100-119 194 194 190 338 338 332 F F F <--- 6 120-139 194 194 187 338 338 327 F F F <--- 7 140-159 187 187 187 327 327 327 F F F 8 160-179 194 194 192 337 337 334 F F F <--- 9 180-199 188 188 188 328 328 328 T T F <--- 10 200-219 192 192 192 335 335 335 T T F <--- 11 220-239 192 192 192 334 334 334 T T F <--- 12 240-259 194 194 191 338 338 333 F F F <--- 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 192 335 335 335 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 192 192 192 335 335 335 F F F 18 360-379 193 193 193 336 336 336 F F F 19 380-399 191 191 191 333 333 333 T T F <--- 20 400-419 194 194 188 338 338 329 T T F <--- 21 420-439 194 194 179 338 338 312 F F T <--- 22 440-459 194 194 186 338 338 325 F F F <--- 23 460-479 190 190 190 331 331 331 T T F <--- 24 480-499 192 192 192 335 335 335 F F F 25 500-519 192 192 192 335 335 335 F F F 26 520-539 194 194 189 338 338 330 F F F <--- 27 540-559 194 194 191 338 338 333 F F F <--- 28 560-579 191 191 191 333 333 333 F F F 29 580-599 194 194 147 338 338 254 F F F <--- Max gen: 194, Max flush: 338 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 194 194 194 337 337 337 F F F 3 060-079 190 190 190 331 331 331 F F F 4 080-099 192 192 192 335 335 335 F F F 5 100-119 194 194 194 338 338 338 F F F 6 120-139 194 194 194 338 338 338 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 194 194 194 337 337 337 F F F 9 180-199 195 195 195 339 339 339 F F F 10 200-219 195 195 195 339 339 339 F F F 11 220-239 195 195 195 339 339 339 F F F 12 240-259 194 194 194 338 338 338 F F F 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 192 335 335 335 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 192 192 192 335 335 335 F F F 18 360-379 193 193 193 336 336 336 F F F 19 380-399 195 195 195 339 339 339 F F F 20 400-419 195 195 195 339 339 339 F F F 21 420-439 194 194 194 338 338 338 F F F 22 440-459 194 194 194 338 338 338 F F F 23 460-479 195 195 195 339 339 339 F F F 24 480-499 192 192 192 335 335 335 F F F 25 500-519 192 192 192 335 335 335 F F F 26 520-539 194 194 194 338 338 338 F F F 27 540-559 194 194 194 338 338 338 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 194 194 194 338 338 338 F F F Max gen: 195, Max flush: 339 Begin loop 98 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.302168679Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.313759735Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.31407838Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314116517Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314128863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314146501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"Crucible c01db826-30fd-4731-b5ea-15607500c86a has session id: 2b22769b-c080-4309-bf87-baffd6c3cdbf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314481443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Upstairs opts: Upstairs UUID: c01db826-30fd-4731-b5ea-15607500c86a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314507195Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Crucible stats registered with UUID: c01db826-30fd-4731-b5ea-15607500c86a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314521992Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:48.314547033Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314725474Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314749335Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314763852Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"1","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314924405Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314941362Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"c01db826-30fd-4731-b5ea-15607500c86a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314964013Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314976779Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.314990895Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"1","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315005151Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315019348Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315039097Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315053714Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"1","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"ds_connection connected from Ok(127.0.0.1:63794)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315082227Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"1","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"ds_connection connected from Ok(127.0.0.1:58325)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315097423Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"ds_connection connected from Ok(127.0.0.1:52020)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315201681Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"io task","client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315717166Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"1","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.315764017Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.31583574Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316075869Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316115627Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[0]R flush_numbers[0..12]: [336, 333, 337, 331, 335, 338, 338, 327, 337, 339, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.31621122Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[0]R generation[0..12]: [193, 191, 194, 190, 192, 194, 194, 187, 194, 195, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316228398Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316244906Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[1]R flush_numbers[0..12]: [336, 333, 337, 331, 335, 338, 338, 327, 337, 339, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316260793Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[1]R generation[0..12]: [193, 191, 194, 190, 192, 194, 194, 187, 194, 195, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.31627641Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316292657Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[2]R flush_numbers[0..12]: [336, 333, 337, 331, 335, 338, 338, 327, 337, 339, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316308535Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[2]R generation[0..12]: [193, 191, 194, 190, 192, 194, 194, 187, 194, 195, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316324192Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316340279Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2026-03-14T05:07:48.316353425Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316383169Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Next flush: 340","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316396835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316410211Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"":"downstairs","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316424387Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316439084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"0","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316453Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"1","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316469818Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"client":"2","":"downstairs","session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316484015Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316497491Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"c01db826-30fd-4731-b5ea-15607500c86a is now active with session: 2b22769b-c080-4309-bf87-baffd6c3cdbf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316512417Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316524863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992,"session_id":"2b22769b-c080-4309-bf87-baffd6c3cdbf"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316538079Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} {"msg":"Activated sub_volume c01db826-30fd-4731-b5ea-15607500c86a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:48.316549594Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 473 len 8192 data: 13 11 02/30 Read block 272 len 8192 03/30 Read block 251 len 8192 04/30 Read block 386 len 4096 05/30 Read block 437 len 36864 06/30 Read block 477 len 16384 07/30 Write block 370 len 4096 data: 16 08/30 Write block 501 len 24576 data: 15 14 14 15 17 16 09/30 Read block 227 len 36864 10/30 Write block 186 len 8192 data: 19 20 11/30 Read block 461 len 28672 12/30 Write block 358 len 40960 data: 19 17 17 14 13 14 15 15 12 13 13/30 Write block 340 len 8192 data: 18 19 14/30 Read block 264 len 36864 15/30 Read block 419 len 28672 16/30 Read block 378 len 36864 17/30 Read block 195 len 12288 18/30 Read block 575 len 28672 19/30 Write block 589 len 20480 data: 7 7 7 8 10 20/30 Read block 139 len 24576 21/30 Read block 71 len 12288 22/30 Read block 58 len 40960 23/30 Read block 324 len 24576 24/30 Write block 580 len 40960 data: 21 20 19 18 17 15 12 8 8 8 25/30 Read block 492 len 16384 26/30 Write block 472 len 20480 data: 12 14 12 13 11 27/30 Write block 86 len 24576 data: 16 16 14 17 21 22 28/30 Write block 79 len 4096 data: 13 29/30 Write block 438 len 20480 data: 23 20 18 12 11 30/30 Write block 256 len 16384 data: 19 17 16 14 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 2 Done Done Sent false 1002 Acked Read 2 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:48.327816813Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":1992} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 194 194 194 337 337 337 F F F 3 060-079 190 190 190 331 331 331 T T F <--- 4 080-099 192 192 192 335 335 335 T T F <--- 5 100-119 194 194 194 338 338 338 F F F 6 120-139 194 194 194 338 338 338 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 194 194 194 337 337 337 F F F 9 180-199 195 195 195 339 339 339 T T F <--- 10 200-219 195 195 195 339 339 339 F F F 11 220-239 195 195 195 339 339 339 F F F 12 240-259 194 194 194 338 338 338 T T F <--- 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 192 335 335 335 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 192 192 192 335 335 335 T T F <--- 18 360-379 193 193 193 336 336 336 T T F <--- 19 380-399 195 195 195 339 339 339 F F F 20 400-419 195 195 195 339 339 339 F F F 21 420-439 194 194 194 338 338 338 T T F <--- 22 440-459 194 194 194 338 338 338 T T F <--- 23 460-479 195 195 195 339 339 339 T T T 24 480-499 192 192 192 335 335 335 F F F 25 500-519 192 192 192 335 335 335 T T F <--- 26 520-539 194 194 194 338 338 338 F F F 27 540-559 194 194 194 338 338 338 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 194 194 194 338 338 338 T T F <--- Max gen: 195, Max flush: 339 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 191 191 333 333 333 F F F 2 040-059 194 194 194 337 337 337 F F F 3 060-079 196 196 196 340 340 340 F F F 4 080-099 196 196 196 340 340 340 F F F 5 100-119 194 194 194 338 338 338 F F F 6 120-139 194 194 194 338 338 338 F F F 7 140-159 187 187 187 327 327 327 F F F 8 160-179 194 194 194 337 337 337 F F F 9 180-199 196 196 196 340 340 340 F F F 10 200-219 195 195 195 339 339 339 F F F 11 220-239 195 195 195 339 339 339 F F F 12 240-259 196 196 196 340 340 340 F F F 13 260-279 190 190 190 331 331 331 F F F 14 280-299 192 192 192 335 335 335 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 196 196 196 340 340 340 F F F 18 360-379 196 196 196 340 340 340 F F F 19 380-399 195 195 195 339 339 339 F F F 20 400-419 195 195 195 339 339 339 F F F 21 420-439 196 196 196 340 340 340 F F F 22 440-459 196 196 196 340 340 340 F F F 23 460-479 196 196 196 340 340 340 F F F 24 480-499 192 192 192 335 335 335 F F F 25 500-519 196 196 196 340 340 340 F F F 26 520-539 194 194 194 338 338 338 F F F 27 540-559 194 194 194 338 338 338 F F F 28 560-579 191 191 191 333 333 333 F F F 29 580-599 196 196 196 340 340 340 F F F Max gen: 196, Max flush: 340 Begin loop 99 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.656030435Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:52.666666076Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.6667193Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.676903485Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677190436Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"b82d0231de26c5ac59ee5d7744e2bafbe7fed1d9\",\n git_commit_timestamp: \"2026-03-14T04:42:34.000000000Z\",\n git_branch: \"renovate/httptest-0.x\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677229944Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67724238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67726433Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"Crucible 9687887b-2a84-4a2e-882e-978e91db4136 has session id: e2e73ddc-08bb-485d-978c-d2022454bb6a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677605835Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Upstairs opts: Upstairs UUID: 9687887b-2a84-4a2e-882e-978e91db4136, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677640561Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Crucible stats registered with UUID: 9687887b-2a84-4a2e-882e-978e91db4136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677656368Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:52.677669374Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677803605Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677839321Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677855238Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.677977084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67799092Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"9687887b-2a84-4a2e-882e-978e91db4136 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678014501Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678027407Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678041443Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67805545Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678069286Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678086404Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67810084Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:40284)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678116457Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:42010)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67825506Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:35121)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678282863Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"io task","client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678807662Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678830162Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.678902205Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67917625Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67921987Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[0]R flush_numbers[0..12]: [336, 333, 337, 340, 340, 338, 338, 327, 337, 340, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679293433Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[0]R generation[0..12]: [193, 191, 194, 196, 196, 194, 194, 187, 194, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679311001Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679328329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[1]R flush_numbers[0..12]: [336, 333, 337, 340, 340, 338, 338, 327, 337, 340, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679344216Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[1]R generation[0..12]: [193, 191, 194, 196, 196, 194, 194, 187, 194, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679362875Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679379472Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[2]R flush_numbers[0..12]: [336, 333, 337, 340, 340, 338, 338, 327, 337, 340, 339, 339]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679395329Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[2]R generation[0..12]: [193, 191, 194, 196, 196, 194, 194, 187, 194, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679411447Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679428224Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Max found gen is 197","v":0,Wait for a query_work_queue command to finish before sending IO "name":"crucible","level":30,"time":"2026-03-14T05:07:52.679447463Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Generation requested: 198 >= found:197","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67946171Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Next flush: 341","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679475256Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679488572Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"":"downstairs","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679502778Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679515564Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"0","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679529591Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"1","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679543347Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"client":"2","":"downstairs","session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679559824Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67957343Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"9687887b-2a84-4a2e-882e-978e91db4136 is now active with session: e2e73ddc-08bb-485d-978c-d2022454bb6a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679590138Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679603274Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000,"session_id":"e2e73ddc-08bb-485d-978c-d2022454bb6a"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.67961664Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} {"msg":"Activated sub_volume 9687887b-2a84-4a2e-882e-978e91db4136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:52.679628145Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 28 len 24576 data: 9 9 9 13 11 12 02/30 Write block 420 len 16384 data: 13 13 15 17 03/30 Read block 93 len 20480 04/30 Write block 166 len 16384 data: 16 17 17 19 05/30 Write block 371 len 16384 data: 14 15 13 15 06/30 Flush 07/30 Write block 557 len 28672 data: 13 12 10 9 11 12 11 08/30 Write block 178 len 32768 data: 15 10 12 11 14 16 15 17 09/30 Read block 2 len 12288 10/30 Read block 225 len 28672 11/30 Read block 137 len 36864 12/30 Write block 256 len 12288 data: 20 18 17 13/30 Read block 450 len 20480 14/30 Read block 447 len 4096 15/30 Read block 485 len 36864 16/30 Write block 519 len 8192 data: 9 10 17/30 Write block 480 len 32768 data: 18 19 17 14 10 11 10 9 18/30 Write block 146 len 28672 data: 16 16 19 18 19 20 21 19/30 Write block 250 len 24576 data: 18 16 21 22 25 22 20/30 Write block 445 len 12288 data: 12 12 10 21/30 Read block 331 len 28672 22/30 Write block 280 len 28672 data: 13 12 15 11 12 14 18 23/30 Read block 361 len 12288 24/30 Read block 166 len 20480 25/30 Write block 568 len 4096 data: 18 26/30 Write block 543 len 4096 data: 16 27/30 Write block 258 len 20480 data: 18 15 15 15 16 28/30 Read block 586 len 24576 29/30 Read block 192 len 36864 30/30 Read block 550 len 4096 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1005)) Some(JobId(1005)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:52.691051585Z","hostname":"w-01KKNBCAY3BP1CJ1ZW7S6ZRS1P","pid":2000} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 191 198 198 333 341 341 T F F <--- 2 040-059 194 194 194 337 337 337 F F F 3 060-079 196 196 196 340 340 340 F F F 4 080-099 196 196 196 340 340 340 F F F 5 100-119 194 194 194 338 338 338 F F F 6 120-139 194 194 194 338 338 338 F F F 7 140-159 187 187 187 327 327 327 F T T <--- 8 160-179 194 198 198 337 341 341 F T T <--- 9 180-199 196 196 196 340 340 340 F T T <--- 10 200-219 195 195 195 339 339 339 F F F 11 220-239 195 195 195 339 339 339 F F F 12 240-259 196 196 196 340 340 340 F T T <--- 13 260-279 190 190 190 331 331 331 F T T <--- 14 280-299 192 192 192 335 335 335 F T T <--- 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 196 196 196 340 340 340 F F F 18 360-379 196 198 198 340 341 341 F F F <--- 19 380-399 195 195 195 339 339 339 F F F 20 400-419 195 195 195 339 339 339 F F F 21 420-439 196 198 198 340 341 341 T F F <--- 22 440-459 196 196 196 340 340 340 F T T <--- 23 460-479 196 196 196 340 340 340 F F F 24 480-499 192 192 192 335 335 335 F T T <--- 25 500-519 196 196 196 340 340 340 F T T <--- 26 520-539 194 194 194 338 338 338 F T T <--- 27 540-559 194 194 194 338 338 338 F T T <--- 28 560-579 191 191 191 333 333 333 F T T <--- 29 580-599 196 196 196 340 340 340 F F F Max gen: 198, Max flush: 341 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 193 193 193 336 336 336 F F F 1 020-039 198 198 198 341 341 341 F F F 2 040-059 194 194 194 337 337 337 F F F 3 060-079 196 196 196 340 340 340 F F F 4 080-099 196 196 196 340 340 340 F F F 5 100-119 194 194 194 338 338 338 F F F 6 120-139 194 194 194 338 338 338 F F F 7 140-159 199 199 199 342 342 342 F F F 8 160-179 199 199 199 342 342 342 F F F 9 180-199 199 199 199 342 342 342 F F F 10 200-219 195 195 195 339 339 339 F F F 11 220-239 195 195 195 339 339 339 F F F 12 240-259 199 199 199 342 342 342 F F F 13 260-279 199 199 199 342 342 342 F F F 14 280-299 199 199 199 342 342 342 F F F 15 300-319 192 192 192 335 335 335 F F F 16 320-339 192 192 192 335 335 335 F F F 17 340-359 196 196 196 340 340 340 F F F 18 360-379 198 198 198 341 341 341 F F F 19 380-399 195 195 195 339 339 339 F F F 20 400-419 195 195 195 339 339 339 F F F 21 420-439 198 198 198 341 341 341 F F F 22 440-459 199 199 199 342 342 342 F F F 23 460-479 196 196 196 340 340 340 F F F 24 480-499 199 199 199 342 342 342 F F F 25 500-519 199 199 199 342 342 342 F F F 26 520-539 199 199 199 342 342 342 F F F 27 540-559 199 199 199 342 342 342 F F F 28 560-579 199 199 199 342 342 342 F F F 29 580-599 196 196 196 340 340 340 F F F Max gen: 199, Max flush: 342 7:37 Test duration Test completed real 7:36.517205382 user 36.018023653 sys 17.433963320 trap 0.015008543 tflt 0.066805035 dflt 0.067046497 kflt 0.003200886 lock 11:56:41.542934397 slp 36:39.753633592 lat 1.365248325 stop 0.034760388