Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.20157912 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.013113001 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.01246997 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1134 1135 1136 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.457523719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.472858118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.47528535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.475379418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.475389147Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.475401067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Crucible 4c16c7f9-cfaf-48ad-90d7-91076fdaf562 has session id: 97e485c9-6eb0-4d05-bd46-c40008523191","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.476423088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Upstairs opts: Upstairs UUID: 4c16c7f9-cfaf-48ad-90d7-91076fdaf562, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.476586984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Crucible stats registered with UUID: 4c16c7f9-cfaf-48ad-90d7-91076fdaf562","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.476597863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:54.476607063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477375751Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.47779211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477803479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477813749Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477822409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"4c16c7f9-cfaf-48ad-90d7-91076fdaf562 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477830649Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477839618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477849308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477858908Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477868488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477878647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477888797Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"ds_connection connected from Ok(127.0.0.1:62635)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477942745Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"ds_connection connected from Ok(127.0.0.1:56353)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477968875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"ds_connection connected from Ok(127.0.0.1:60271)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.477980224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.480070206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.480094245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.480108455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481062488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481073017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481177915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481188734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481199784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481210074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481220113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481230973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481241093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481251072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481262022Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481271432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481281192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481290681Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481302201Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"downstairs","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481312341Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481321351Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.4813313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.48134149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"client":"2","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.48135153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481360759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"4c16c7f9-cfaf-48ad-90d7-91076fdaf562 is now active with session: 97e485c9-6eb0-4d05-bd46-c40008523191","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481370729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481379899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481387489Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"Activated sub_volume 4c16c7f9-cfaf-48ad-90d7-91076fdaf562","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481919624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.481973892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Fill test","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.482444699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} Read and Verify all blocks (0..600 range:false) {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30---------------------------------------------------------------- ,"time":"2026-09-24T22:08:54.522487883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"CLIENT: Tests done. All submitted work has been ACK'd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.522521042Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 {"msg":"CLIENT: Up:0 ds:3 act:3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:54.522655399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 {"msg":"CLIENT: Up:0 ds:0 act:3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.523299387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.523341235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"task":"crutest"} {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523357125Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139} {"msg":"client request queue closed unexpectedly; is the program exiting?","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523423423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client request queue closed unexpectedly; is the program exiting?","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523438153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523448942Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523459552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523469842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.523480601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.523491081Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.523501561Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"97e485c9-6eb0-4d05-bd46-c40008523191"} Begin loop 1 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.548341813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55660447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.556927241Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55695468Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55696328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55697523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Crucible 4a0deced-3437-46a1-97b6-b6f87bb19c66 has session id: 08150b21-86d3-4b8b-9020-14a5ff3d4147","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55733571Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Upstairs opts: Upstairs UUID: 4a0deced-3437-46a1-97b6-b6f87bb19c66, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557360549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Crucible stats registered with UUID: 4a0deced-3437-46a1-97b6-b6f87bb19c66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557370789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.557379878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557546744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557577803Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557589363Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557703899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557718509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"4a0deced-3437-46a1-97b6-b6f87bb19c66 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557753778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557763288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557774937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557784827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557794517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557804637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557837656Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"ds_connection connected from Ok(127.0.0.1:35095)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557900484Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"ds_connection connected from Ok(127.0.0.1:56476)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557925743Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"ds_connection connected from Ok(127.0.0.1:51336)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.557937353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558593664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558626333Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558687572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558922545Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558950004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.558995793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559018812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559032822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559043292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559053691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559064811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559075111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559085441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55909662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55910634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.55911631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559125879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559135469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"":"downstairs","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559145499Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559154429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"0","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559164458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"1","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559174298Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"client":"2","":"downstairs","session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559185948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559195217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"4a0deced-3437-46a1-97b6-b6f87bb19c66 is now active with session: 08150b21-86d3-4b8b-9020-14a5ff3d4147","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559205217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559214417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559222157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"session_id":"08150b21-86d3-4b8b-9020-14a5ff3d4147"} {"msg":"Activated sub_volume 4a0deced-3437-46a1-97b6-b6f87bb19c66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559231396Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559239526Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.559861319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} 01/30 Write block 163 len 8192 data: 2 2 02/30 Read block 232 len 8192 03/30 Read block 129 len 40960 04/30 Read block 531 len 20480 05/30 Read block 460 len 20480 06/30 Flush 07/30 Write block 329 len 8192 data: 2 2 08/30 Write block 588 len 4096 data: 2 09/30 Read block 196 len 28672 10/30 Write block 6 len 16384 data: 2 2 2 2 11/30 Write block 234 len 36864 data: 2 2 2 2 2 2 2 2 2 12/30 Read block 592 len 32768 13/30 Write block 540 len 24576 data: 2 2 2 2 2 2 14/30 Write block 125 len 16384 data: 2 2 2 2 15/30 Read block 37 len 40960 16/30 Read block 273 len 4096 17/30 Read block 545 len 12288 18/30 Read block 259 len 8192 19/30 Read block 145 len 16384 20/30 Read block 256 len 8192 21/30 Write block 306 len 16384 data: 2 2 2 2 22/30 Read block 517 len 4096 23/30 Write block 445 len 16384 data: 2 2 2 2 24/30 Read block 56 len 12288 25/30 Write block 374 len 16384 data: 2 2 2 2 26/30 Write block 535 len 32768 data: 2 2 2 2 2 3 3 3 27/30 Read block 412 len 12288 28/30 Read block 29 len 40960 29/30 Write block 74 len 28672 data: 2 2 2 2 2 2 2 30/30 Write block 11 len 24576 data: 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 26 2 30 Done 28 4 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) None Some(JobId(1005)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:08:58.568259813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:08:58.568447227Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1141,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 T F T <--- 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 T F T <--- 4 080-099 1 1 1 1 1 1 T F T <--- 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 T F T <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 2 1 2 2 1 2 F T F <--- 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 T F T <--- 12 240-259 1 1 1 1 1 1 T F T <--- 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 T F T <--- 16 320-339 1 1 1 1 1 1 T F T <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 T F T <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 T F T <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 T F T <--- 27 540-559 1 1 1 1 1 1 T F T <--- 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 T F T <--- Max gen: 2, Max flush: 2 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 3 3 3 3 3 3 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 3 3 3 3 3 3 F F F 12 240-259 3 3 3 3 3 3 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 3 3 3 3 3 3 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 3 3 3 3 3 3 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 3 3 3 3 3 3 F F F 27 540-559 3 3 3 3 3 3 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 3 3 3 3 3 3 F F F Max gen: 3, Max flush: 3 Begin loop 2 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.95986841Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968357391Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.96838533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.96839435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.96840609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Crucible b2560247-5320-43bf-90a8-c0ffb9b62a85 has session id: 1cddb630-b10d-42c5-9754-94a52269156d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968720851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Upstairs opts: Upstairs UUID: b2560247-5320-43bf-90a8-c0ffb9b62a85, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.96876475Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Crucible stats registered with UUID: b2560247-5320-43bf-90a8-c0ffb9b62a85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968777569Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:02.968786579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968933865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968950704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.968960534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.9691018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.96912621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"b2560247-5320-43bf-90a8-c0ffb9b62a85 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969140259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969149719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969160059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969169878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969179498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969189388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969227807Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"ds_connection connected from Ok(127.0.0.1:35074)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969251776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"ds_connection connected from Ok(127.0.0.1:42964)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969265626Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47522)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969286445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969961896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.969981165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970020984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970292257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970314076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[0]R flush_numbers[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970324546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[0]R generation[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970335466Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970346735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[1]R flush_numbers[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970357065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[1]R generation[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970367215Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970378424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[2]R flush_numbers[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970388774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[2]R generation[0..12]: [3, 1, 1, 3, 3, 1, 3, 1, 2, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970401634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970412623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970422113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970431823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Next flush: 4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970441233Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970450792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"":"downstairs","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970460552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970469452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"0","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970479301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"1","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970488951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"client":"2","":"downstairs","session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970498811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970507851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"b2560247-5320-43bf-90a8-c0ffb9b62a85 is now active with session: 1cddb630-b10d-42c5-9754-94a52269156d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.97051785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.97052676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"session_id":"1cddb630-b10d-42c5-9754-94a52269156d"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.97053579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Activated sub_volume b2560247-5320-43bf-90a8-c0ffb9b62a85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.97054386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970552049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.970866681Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} 01/30 Write block 177 len 4096 data: 2 02/30 Read block 456 len 32768 03/30 Read block 232 len 36864 04/30 Write block 400 len 36864 data: 2 2 2 2 2 2 2 2 2 05/30 Write block 415 len 8192 data: 2 2 06/30 Write block 409 len 24576 data: 2 2 2 2 2 2 07/30 Read block 355 len 28672 08/30 Flush 09/30 Write block 20 len 4096 data: 2 10/30 Flush 11/30 Write block 523 len 16384 data: 2 2 2 2 12/30 Read block 394 len 40960 13/30 Read block 527 len 4096 14/30 Write block 136 len 36864 data: 2 2 2 2 2 2 2 2 2 15/30 Read block 321 len 8192 16/30 Write block 384 len 40960 data: 2 2 2 2 2 2 2 2 2 2 17/30 Write block 248 len 28672 data: 2 2 2 2 2 2 2 18/30 Read block 381 len 28672 19/30 Read block 27 len 24576 20/30 Write block 277 len 12288 data: 2 2 2 21/30 Read block 150 len 28672 22/30 Flush 23/30 Write block 462 len 36864 data: 2 2 2 2 2 2 2 2 2 24/30 Read block 395 len 32768 25/30 Read block 381 len 40960 26/30 Write block 470 len 28672 data: 3 2 2 2 2 2 2 27/30 Write block 381 len 40960 data: 2 2 2 3 3 3 3 3 3 3 28/30 Write block 542 len 28672 data: 4 3 3 3 2 2 2 29/30 Write block 109 len 36864 data: 2 2 2 2 2 2 2 2 2 30/30 Write block 39 len 20480 data: 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 9 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 28 34 Done 27 27 2 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:02.979174547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:02.979348892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1149,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F F F 1 020-039 4 4 1 5 5 1 T T F <--- 2 040-059 1 1 1 1 1 1 T T F <--- 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 1 1 1 1 1 1 T T F <--- 6 120-139 4 4 3 6 6 3 F F F <--- 7 140-159 4 4 1 6 6 1 F F F <--- 8 160-179 4 4 2 4 4 2 F F T <--- 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 3 3 3 3 3 3 F F F 12 240-259 4 4 3 6 6 3 F F F <--- 13 260-279 4 4 1 6 6 1 F F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 3 3 3 3 3 3 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 3 3 3 3 3 3 F F F 19 380-399 4 4 1 6 6 1 T T F <--- 20 400-419 4 4 1 4 4 1 F F F <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 1 1 1 1 1 1 T T F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 4 4 3 6 6 3 F F F <--- 27 540-559 3 3 3 3 3 3 T T F <--- 28 560-579 1 1 1 1 1 1 F F F 29 580-599 3 3 3 3 3 3 F F F Max gen: 4, Max flush: 6 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F F F 1 020-039 5 5 5 7 7 7 F F F 2 040-059 5 5 5 7 7 7 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 5 5 5 7 7 7 F F F 6 120-139 4 4 4 6 6 6 F F F 7 140-159 4 4 4 6 6 6 F F F 8 160-179 4 4 4 4 4 4 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 3 3 3 3 3 3 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 6 6 6 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 3 3 3 3 3 3 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 3 3 3 3 3 3 F F F 19 380-399 5 5 5 7 7 7 F F F 20 400-419 4 4 4 4 4 4 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 5 5 5 7 7 7 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 5 5 5 7 7 7 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 3 3 3 3 3 3 F F F Max gen: 5, Max flush: 7 Begin loop 3 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.225788191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.234387019Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.234686581Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.2347117Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.23472094Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.234733069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Crucible 44596104-a15c-4a78-a329-a3e7a4f66093 has session id: 6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235079059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Upstairs opts: Upstairs UUID: 44596104-a15c-4a78-a329-a3e7a4f66093, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235110929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Crucible stats registered with UUID: 44596104-a15c-4a78-a329-a3e7a4f66093","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235121158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:07.235130328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235298423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235329672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235340842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235495968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235510917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"44596104-a15c-4a78-a329-a3e7a4f66093 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235547366Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235556946Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235566746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235576695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235586315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235596405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235606435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:38039)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235654923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:59161)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235665903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:43202)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.235676383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236356354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236382853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236394982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236681734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236715533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[0]R flush_numbers[0..12]: [3, 7, 7, 3, 3, 7, 6, 6, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236777952Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[0]R generation[0..12]: [3, 5, 5, 3, 3, 5, 4, 4, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236789481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236800871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[1]R flush_numbers[0..12]: [3, 7, 7, 3, 3, 7, 6, 6, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236811371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[1]R generation[0..12]: [3, 5, 5, 3, 3, 5, 4, 4, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.23682182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.23683333Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[2]R flush_numbers[0..12]: [3, 7, 7, 3, 3, 7, 6, 6, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.23684396Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[2]R generation[0..12]: [3, 5, 5, 3, 3, 5, 4, 4, 4, 1, 1, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.23685421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236865219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236874849Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236884779Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Next flush: 8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236894318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236903748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"":"downstairs","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236913968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236924948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"0","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236935307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"1","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236945367Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"client":"2","":"downstairs","session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236955557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236964996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"44596104-a15c-4a78-a329-a3e7a4f66093 is now active with session: 6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236975326Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.236989296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"session_id":"6bc8e13c-68fb-47d6-81ba-64f9dca0b5c2"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.237016635Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Activated sub_volume 44596104-a15c-4a78-a329-a3e7a4f66093","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.237024915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.237033075Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.237348946Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} 01/30 Write block 179 len 24576 data: 2 2 2 2 2 2 02/30 Write block 552 len 36864 data: 2 2 2 2 2 2 2 2 2 03/30 Read block 141 len 12288 04/30 Read block 127 len 12288 05/30 Write block 265 len 32768 data: 2 2 2 2 2 2 2 2 06/30 Write block 582 len 40960 data: 2 2 2 2 2 2 3 2 2 2 07/30 Read block 29 len 12288 08/30 Write block 329 len 12288 data: 3 3 2 09/30 Read block 310 len 28672 10/30 Read block 210 len 36864 11/30 Write block 387 len 12288 data: 4 4 4 12/30 Write block 44 len 40960 data: 2 2 2 2 2 2 2 2 2 2 13/30 Read block 281 len 4096 14/30 Flush 15/30 Write block 358 len 28672 data: 2 2 2 2 2 2 2 16/30 Flush 17/30 Write block 546 len 12288 data: 3 3 3 18/30 Read block 244 len 32768 19/30 Write block 161 len 8192 data: 2 2 20/30 Flush 21/30 Write block 443 len 16384 data: 2 2 3 3 22/30 Write block 382 len 32768 data: 3 3 4 4 4 5 5 5 23/30 Read block 20 len 4096 24/30 Write block 209 len 4096 data: 2 25/30 Write block 319 len 4096 data: 2 26/30 Write block 568 len 12288 data: 2 2 2 27/30 Read block 300 len 32768 28/30 Read block 39 len 20480 29/30 Read block 446 len 24576 30/30 Write block 411 len 28672 data: 3 3 3 3 3 3 2 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 1 1 25 Done 7 29 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:07.245813958Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:07.246010482Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1157,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F F F 1 020-039 5 5 5 7 7 7 F F F 2 040-059 5 6 6 7 8 8 F F F <--- 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 5 5 5 7 7 7 F F F 6 120-139 4 4 4 6 6 6 F F F 7 140-159 4 4 4 6 6 6 F F F 8 160-179 4 6 6 4 10 10 T F F <--- 9 180-199 1 6 6 1 8 8 T F F <--- 10 200-219 1 1 1 1 1 1 F T T <--- 11 220-239 3 3 3 3 3 3 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 6 6 6 8 8 T F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F T T <--- 16 320-339 3 6 6 3 8 8 F F F <--- 17 340-359 1 6 6 1 9 9 F F F <--- 18 360-379 3 6 6 3 9 9 F F F <--- 19 380-399 5 6 6 7 8 8 F T T <--- 20 400-419 4 4 4 4 4 4 F T T <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 3 3 3 3 3 3 F T T <--- 23 460-479 5 5 5 7 7 7 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 5 6 6 7 10 10 T F F <--- 28 560-579 1 6 6 1 8 8 T T T <--- 29 580-599 3 6 6 3 8 8 T F F <--- Max gen: 6, Max flush: 10 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F F F 1 020-039 5 5 5 7 7 7 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 5 5 5 7 7 7 F F F 6 120-139 4 4 4 6 6 6 F F F 7 140-159 4 4 4 6 6 6 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 3 3 3 3 3 3 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 7 7 7 11 11 11 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 6 6 6 9 9 9 F F F 18 360-379 6 6 6 9 9 9 F F F 19 380-399 7 7 7 11 11 11 F F F 20 400-419 7 7 7 11 11 11 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 7 7 7 11 11 11 F F F 23 460-479 5 5 5 7 7 7 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 7 7 7 11 11 11 F F F 29 580-599 6 6 6 8 8 8 F F F Max gen: 7, Max flush: 11 Begin loop 4 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.507924696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.516372928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.51665515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.516683349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.516692099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.516706999Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Crucible 91139df5-0c00-48fa-aa7c-d9cd79ed834c has session id: 2b580aee-8649-4c68-8c26-1a3e0823de41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517067979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Upstairs opts: Upstairs UUID: 91139df5-0c00-48fa-aa7c-d9cd79ed834c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517113787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Crucible stats registered with UUID: 91139df5-0c00-48fa-aa7c-d9cd79ed834c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517138897Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:11.517148836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517268483Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517298302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517309192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517416449Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517433408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"91139df5-0c00-48fa-aa7c-d9cd79ed834c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517457248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517466777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517476467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517486187Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517495767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517508096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517518166Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"ds_connection connected from Ok(127.0.0.1:33261)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517596344Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"ds_connection connected from Ok(127.0.0.1:49404)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517622153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"ds_connection connected from Ok(127.0.0.1:46648)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.517633863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518301604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518321783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518332903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518611635Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518621235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[0]R flush_numbers[0..12]: [3, 7, 8, 3, 3, 7, 6, 6, 10, 8, 11, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518649814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[0]R generation[0..12]: [3, 5, 6, 3, 3, 5, 4, 4, 6, 6, 7, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518660794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518672194Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[1]R flush_numbers[0..12]: [3, 7, 8, 3, 3, 7, 6, 6, 10, 8, 11, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518682783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[1]R generation[0..12]: [3, 5, 6, 3, 3, 5, 4, 4, 6, 6, 7, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518693173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518706883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[2]R flush_numbers[0..12]: [3, 7, 8, 3, 3, 7, 6, 6, 10, 8, 11, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518717322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[2]R generation[0..12]: [3, 5, 6, 3, 3, 5, 4, 4, 6, 6, 7, 3]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518727882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518739152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518748771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518758801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Next flush: 12","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518768491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518778091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"":"downstairs","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.51878809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.51879706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"0","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.51880689Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"1","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.5188167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"client":"2","":"downstairs","session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518827209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518836589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"91139df5-0c00-48fa-aa7c-d9cd79ed834c is now active with session: 2b580aee-8649-4c68-8c26-1a3e0823de41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518846789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518857288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"session_id":"2b580aee-8649-4c68-8c26-1a3e0823de41"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518866638Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Activated sub_volume 91139df5-0c00-48fa-aa7c-d9cd79ed834c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518874788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.518883118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.519222228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} 01/30 Write block 317 len 12288 data: 2 2 3 02/30 Read block 49 len 36864 03/30 Write block 220 len 32768 data: 2 2 2 2 2 2 2 2 04/30 Write block 328 len 40960 data: 2 4 4 3 2 2 2 2 2 2 05/30 Read block 26 len 32768 06/30 Read block 485 len 28672 07/30 Flush 08/30 Write block 115 len 20480 data: 3 3 3 2 2 09/30 Write block 123 len 8192 data: 2 2 10/30 Write block 554 len 16384 data: 3 3 3 3 11/30 Read block 77 len 20480 12/30 Read block 517 len 8192 13/30 Write block 66 len 4096 data: 2 14/30 Read block 536 len 8192 15/30 Read block 566 len 32768 16/30 Read block 290 len 40960 17/30 Write block 243 len 24576 data: 2 2 2 2 2 3 18/30 Read block 464 len 28672 19/30 Write block 383 len 32768 data: 4 5 5 5 6 6 6 4 20/30 Read block 226 len 32768 21/30 Read block 152 len 12288 22/30 Read block 421 len 16384 23/30 Read block 110 len 28672 24/30 Read block 92 len 24576 25/30 Write block 19 len 24576 data: 2 3 2 2 2 2 26/30 Write block 9 len 12288 data: 3 2 3 27/30 Write block 511 len 16384 data: 2 2 2 2 28/30 Read block 461 len 28672 29/30 Write block 246 len 4096 data: 3 30/30 Write block 323 len 20480 data: 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 2 28 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:11.527620592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:11.527803807Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1165,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 3 3 3 F T T <--- 1 020-039 5 5 5 7 7 7 F T T <--- 2 040-059 6 6 6 8 8 8 F F F 3 060-079 3 3 3 3 3 3 F T T <--- 4 080-099 3 3 3 3 3 3 F F F 5 100-119 5 5 5 7 7 7 F T T <--- 6 120-139 4 4 4 6 6 6 F T T <--- 7 140-159 4 4 4 6 6 6 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 3 8 8 3 12 12 F F F <--- 12 240-259 4 4 4 6 6 6 F T T <--- 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 7 8 8 11 12 12 T F F <--- 16 320-339 6 8 8 8 12 12 F T T <--- 17 340-359 6 6 6 9 9 9 F F F 18 360-379 6 6 6 9 9 9 F F F 19 380-399 7 7 7 11 11 11 F T T <--- 20 400-419 7 7 7 11 11 11 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 7 7 7 11 11 11 F F F 23 460-479 5 5 5 7 7 7 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F T T <--- 26 520-539 4 4 4 6 6 6 F F F 27 540-559 6 6 6 10 10 10 F T T <--- 28 560-579 7 7 7 11 11 11 F F F 29 580-599 6 6 6 8 8 8 F F F Max gen: 8, Max flush: 12 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 9 9 9 13 13 13 F F F 1 020-039 9 9 9 13 13 13 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 9 9 9 13 13 13 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 9 9 9 13 13 13 F F F 7 140-159 4 4 4 6 6 6 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 8 8 8 12 12 12 F F F 12 240-259 9 9 9 13 13 13 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 8 8 8 12 12 12 F F F 16 320-339 9 9 9 13 13 13 F F F 17 340-359 6 6 6 9 9 9 F F F 18 360-379 6 6 6 9 9 9 F F F 19 380-399 9 9 9 13 13 13 F F F 20 400-419 7 7 7 11 11 11 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 7 7 7 11 11 11 F F F 23 460-479 5 5 5 7 7 7 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 9 9 9 13 13 13 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 9 9 9 13 13 13 F F F 28 560-579 7 7 7 11 11 11 F F F 29 580-599 6 6 6 8 8 8 F F F Max gen: 9, Max flush: 13 Begin loop 5 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.786818192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795038001Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795336262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795360562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795369252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795381151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Crucible 4774ed43-c686-4e18-a512-36c5da0420fe has session id: e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795744371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Upstairs opts: Upstairs UUID: 4774ed43-c686-4e18-a512-36c5da0420fe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.79577357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Crucible stats registered with UUID: 4774ed43-c686-4e18-a512-36c5da0420fe","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.79578491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:15.79579425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795967105Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.795997604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796008684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796116281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.7961335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"4774ed43-c686-4e18-a512-36c5da0420fe active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796157799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796167429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796177329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796187259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796197058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796207228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796217258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:55608)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796318485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45682)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796330135Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:37332)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.796340654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797059284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797084733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797096243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797387185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797397445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[0]R flush_numbers[0..12]: [13, 13, 8, 13, 3, 13, 13, 6, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797434093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[0]R generation[0..12]: [9, 9, 6, 9, 3, 9, 9, 4, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797445253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797456533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[1]R flush_numbers[0..12]: [13, 13, 8, 13, 3, 13, 13, 6, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797467153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[1]R generation[0..12]: [9, 9, 6, 9, 3, 9, 9, 4, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797477502Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797488702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[2]R flush_numbers[0..12]: [13, 13, 8, 13, 3, 13, 13, 6, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797499222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[2]R generation[0..12]: [9, 9, 6, 9, 3, 9, 9, 4, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797509481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797520631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797530111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797539901Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Next flush: 14","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.79755091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.79756091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"":"downstairs","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.79757072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797579579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"0","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797589599Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"1","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797599369Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"client":"2","":"downstairs","session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797609339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797618558Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"4774ed43-c686-4e18-a512-36c5da0420fe is now active with session: e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797628678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797638018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797645708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"session_id":"e1f7f8ab-ba49-4c99-853d-dea0bf1d7f3d"} {"msg":"Activated sub_volume 4774ed43-c686-4e18-a512-36c5da0420fe","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797655067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797663457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.797969498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} 01/30 Write block 365 len 40960 data: 2 2 2 2 2 2 2 2 2 3 02/30 Read block 44 len 16384 03/30 Write block 42 len 24576 data: 3 3 3 3 3 3 04/30 Read block 464 len 40960 05/30 Write block 579 len 40960 data: 2 2 2 3 3 3 3 3 3 4 06/30 Flush 07/30 Write block 151 len 12288 data: 2 2 2 08/30 Write block 543 len 4096 data: 4 09/30 Write block 375 len 28672 data: 3 3 3 2 2 2 3 10/30 Read block 71 len 12288 11/30 Write block 420 len 28672 data: 2 2 2 2 2 2 2 12/30 Write block 144 len 8192 data: 3 2 13/30 Write block 427 len 32768 data: 2 2 2 2 2 2 2 2 14/30 Read block 259 len 8192 15/30 Write block 358 len 8192 data: 3 3 16/30 Write block 453 len 32768 data: 2 2 2 2 2 2 2 2 17/30 Read block 140 len 28672 18/30 Write block 19 len 36864 data: 3 4 3 3 3 3 2 2 2 19/30 Flush 20/30 Write block 450 len 36864 data: 2 2 2 3 3 3 3 3 3 21/30 Read block 408 len 24576 22/30 Read block 84 len 32768 23/30 Write block 560 len 36864 data: 3 2 2 2 2 2 2 2 3 24/30 Read block 494 len 20480 25/30 Write block 27 len 32768 data: 3 2 2 2 2 2 2 2 26/30 Flush 27/30 Write block 149 len 40960 data: 2 2 3 3 3 2 2 2 2 2 28/30 Read block 488 len 28672 29/30 Read block 507 len 32768 30/30 Read block 127 len 36864 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Read 4 Done Done Sent false 1002 Acked Write 6 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 30 31 Done 29 30 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:15.807432232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:15.807618817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1173,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 9 15 15 13 F F F <--- 1 020-039 10 10 9 16 16 13 F F F <--- 2 040-059 10 10 6 14 14 8 F F F <--- 3 060-079 9 9 9 13 13 13 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 9 9 9 13 13 13 F F F 7 140-159 10 10 4 15 15 6 T T F <--- 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 8 8 8 12 12 12 F F F 12 240-259 9 9 9 13 13 13 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 8 8 8 12 12 12 F F F 16 320-339 9 9 9 13 13 13 F F F 17 340-359 10 10 6 15 15 9 F F F <--- 18 360-379 10 10 6 15 15 9 F F F <--- 19 380-399 10 10 9 15 15 13 F F F <--- 20 400-419 7 7 7 11 11 11 F F F 21 420-439 10 10 1 15 15 1 F F F <--- 22 440-459 10 10 7 16 16 11 F F F <--- 23 460-479 10 10 5 15 15 7 F F F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 9 9 9 13 13 13 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 10 10 9 15 15 13 F F F <--- 28 560-579 10 10 7 16 16 11 F F F <--- 29 580-599 10 10 6 14 14 8 F F F <--- Max gen: 10, Max flush: 16 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 15 15 15 F F F 1 020-039 10 10 10 16 16 16 F F F 2 040-059 10 10 10 14 14 14 F F F 3 060-079 9 9 9 13 13 13 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 9 9 9 13 13 13 F F F 7 140-159 11 11 11 17 17 17 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 8 8 8 12 12 12 F F F 12 240-259 9 9 9 13 13 13 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 8 8 8 12 12 12 F F F 16 320-339 9 9 9 13 13 13 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 10 10 10 15 15 15 F F F 20 400-419 7 7 7 11 11 11 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 10 10 10 16 16 16 F F F 23 460-479 10 10 10 15 15 15 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 9 9 9 13 13 13 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 10 10 10 14 14 14 F F F Max gen: 11, Max flush: 17 Begin loop 6 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.057521468Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.065751016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066050058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066077657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066086277Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066098157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Crucible 924239f3-65a6-471e-a47a-1d1b6b28e9bd has session id: 006207c2-8402-401b-b822-8fa8f619b4cb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066446177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Upstairs opts: Upstairs UUID: 924239f3-65a6-471e-a47a-1d1b6b28e9bd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066474586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Crucible stats registered with UUID: 924239f3-65a6-471e-a47a-1d1b6b28e9bd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066485766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:20.066495225Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066645761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.06667707Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.0666888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066802417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066820776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"924239f3-65a6-471e-a47a-1d1b6b28e9bd active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066851295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066860925Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066872975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066882905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066892764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066902984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066913104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:56933)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066947823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:64896)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066969462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:39397)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.066984682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.067653673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.067682392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.067735761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.067950685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.067969704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[0]R flush_numbers[0..12]: [15, 16, 14, 13, 3, 13, 13, 17, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068040262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[0]R generation[0..12]: [10, 10, 10, 9, 3, 9, 9, 11, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068051412Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068065181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[1]R flush_numbers[0..12]: [15, 16, 14, 13, 3, 13, 13, 17, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068075781Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[1]R generation[0..12]: [10, 10, 10, 9, 3, 9, 9, 11, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068086161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.06809708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[2]R flush_numbers[0..12]: [15, 16, 14, 13, 3, 13, 13, 17, 10, 8, 11, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.06810791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[2]R generation[0..12]: [10, 10, 10, 9, 3, 9, 9, 11, 6, 6, 7, 8]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.06811818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068129269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068138959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068148709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068158139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068167578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"":"downstairs","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068177498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068186438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"0","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068196408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"1","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068206327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"client":"2","":"downstairs","session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068217707Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068226827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"924239f3-65a6-471e-a47a-1d1b6b28e9bd is now active with session: 006207c2-8402-401b-b822-8fa8f619b4cb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068237006Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068245676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"session_id":"006207c2-8402-401b-b822-8fa8f619b4cb"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068254896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Activated sub_volume 924239f3-65a6-471e-a47a-1d1b6b28e9bd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068262936Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068270986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.068581247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} 01/30 Write block 321 len 40960 data: 2 2 3 3 3 3 3 3 5 5 02/30 Read block 584 len 28672 03/30 Read block 24 len 36864 04/30 Read block 72 len 8192 05/30 Flush 06/30 Write block 229 len 28672 data: 2 2 2 2 2 3 3 07/30 Read block 22 len 12288 08/30 Write block 57 len 28672 data: 2 2 2 2 2 2 2 09/30 Write block 66 len 20480 data: 3 2 2 2 2 10/30 Read block 289 len 28672 11/30 Write block 35 len 4096 data: 2 12/30 Read block 142 len 16384 13/30 Flush 14/30 Write block 275 len 4096 data: 2 15/30 Write block 38 len 20480 data: 2 3 3 3 4 16/30 Write block 209 len 16384 data: 3 2 2 2 17/30 Read block 5 len 36864 18/30 Write block 320 len 20480 data: 2 3 3 4 4 19/30 Write block 394 len 36864 data: 2 2 2 2 2 2 3 3 3 20/30 Read block 186 len 20480 21/30 Write block 286 len 8192 data: 2 2 22/30 Read block 276 len 20480 23/30 Write block 7 len 40960 data: 3 3 4 3 4 3 3 3 3 3 24/30 Read block 165 len 32768 25/30 Flush 26/30 Write block 63 len 32768 data: 3 2 2 4 3 3 3 3 27/30 Write block 127 len 16384 data: 3 3 2 2 28/30 Write block 275 len 4096 data: 3 29/30 Read block 11 len 32768 30/30 Read block 523 len 20480 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 30 1 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:20.077603393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:20.077780548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1181,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 12 10 12 20 15 20 F F F <--- 1 020-039 12 10 12 20 16 20 F F F <--- 2 040-059 12 10 12 20 14 20 F F F <--- 3 060-079 12 9 12 19 13 19 T F T <--- 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 9 9 9 13 13 13 T F T <--- 7 140-159 11 11 11 17 17 17 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 12 7 12 20 11 20 F F F <--- 11 220-239 12 8 12 19 12 19 F F F <--- 12 240-259 9 9 9 13 13 13 F F F 13 260-279 12 6 12 20 8 20 T F T <--- 14 280-299 12 1 12 20 1 20 F F F <--- 15 300-319 8 8 8 12 12 12 F F F 16 320-339 12 9 12 20 13 20 F T F <--- 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 12 10 12 20 15 20 F F F <--- 20 400-419 12 7 12 20 11 20 F F F <--- 21 420-439 10 10 10 15 15 15 F F F 22 440-459 10 10 10 16 16 16 F F F 23 460-479 10 10 10 15 15 15 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 9 9 9 13 13 13 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 10 10 10 14 14 14 F F F Max gen: 12, Max flush: 20 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 12 12 12 20 20 20 F F F 1 020-039 12 12 12 20 20 20 F F F 2 040-059 12 12 12 20 20 20 F F F 3 060-079 13 13 13 21 21 21 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 11 11 11 17 17 17 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 12 12 12 20 20 20 F F F 11 220-239 12 12 12 19 19 19 F F F 12 240-259 9 9 9 13 13 13 F F F 13 260-279 13 13 13 21 21 21 F F F 14 280-299 12 12 12 20 20 20 F F F 15 300-319 8 8 8 12 12 12 F F F 16 320-339 12 12 12 20 20 20 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 12 12 12 20 20 20 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 10 10 10 16 16 16 F F F 23 460-479 10 10 10 15 15 15 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 9 9 9 13 13 13 F F F 26 520-539 4 4 4 6 6 6 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 10 10 10 14 14 14 F F F Max gen: 13, Max flush: 21 Begin loop 7 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.331142462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:24.338277332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.3383357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345275535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345554737Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345590416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345599956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345612185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Crucible a9b02439-3bd2-4f97-81db-865ad32bdf60 has session id: bee469c9-fcf5-4020-89b0-e3ef1434aba6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345935256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Upstairs opts: Upstairs UUID: a9b02439-3bd2-4f97-81db-865ad32bdf60, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345964335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Crucible stats registered with UUID: a9b02439-3bd2-4f97-81db-865ad32bdf60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.345975225Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:24.345984455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346128131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34615859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34616964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346297726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346311746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"a9b02439-3bd2-4f97-81db-865ad32bdf60 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346335355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346344735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346354554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346364344Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346373994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346384084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346396843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"ds_connection connected from Ok(127.0.0.1:51396)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346416573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"ds_connection connected from Ok(127.0.0.1:54826)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.346427072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"ds_connection connected from Ok(127.0.0.1:58506)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34651677Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347085614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347105123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34721632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347452304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347474813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[0]R flush_numbers[0..12]: [20, 20, 20, 21, 3, 13, 21, 17, 10, 8, 20, 19]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34756637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[0]R generation[0..12]: [12, 12, 12, 13, 3, 9, 13, 11, 6, 6, 12, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34757811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.34758921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[1]R flush_numbers[0..12]: [20, 20, 20, 21, 3, 13, 21, 17, 10, 8, 20, 19]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347599679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[1]R generation[0..12]: [12, 12, 12, 13, 3, 9, 13, 11, 6, 6, 12, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347610169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347621269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[2]R flush_numbers[0..12]: [20, 20, 20, 21, 3, 13, 21, 17, 10, 8, 20, 19]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347633969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[2]R generation[0..12]: [12, 12, 12, 13, 3, 9, 13, 11, 6, 6, 12, 12]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347644578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347655578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347665068Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347675017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Next flush: 22","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347684607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347694117Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"":"downstairs","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347703967Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347712806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"0","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347722386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"1","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347732126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"client":"2","":"downstairs","session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347741965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347751155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"a9b02439-3bd2-4f97-81db-865ad32bdf60 is now active with session: bee469c9-fcf5-4020-89b0-e3ef1434aba6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347761305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347769895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"session_id":"bee469c9-fcf5-4020-89b0-e3ef1434aba6"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347780364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Activated sub_volume a9b02439-3bd2-4f97-81db-865ad32bdf60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347788354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.347796314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.348167324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} 01/30 Write block 259 len 28672 data: 2 2 2 2 2 2 3 02/30 Flush 03/30 Write block 518 len 40960 data: 2 2 2 2 2 3 3 3 3 2 04/30 Write block 408 len 8192 data: 3 3 05/30 Read block 457 len 40960 06/30 Read block 203 len 8192 07/30 Read block 485 len 36864 08/30 Read block 393 len 16384 09/30 Read block 379 len 8192 10/30 Read block 181 len 28672 11/30 Flush 12/30 Write block 298 len 36864 data: 2 2 2 2 2 2 2 2 3 13/30 Read block 60 len 8192 14/30 Write block 212 len 40960 data: 3 2 2 2 2 2 2 2 3 3 15/30 Write block 584 len 24576 data: 4 4 4 4 5 3 16/30 Read block 460 len 24576 17/30 Write block 449 len 36864 data: 2 3 3 3 4 4 4 4 4 18/30 Read block 527 len 32768 19/30 Write block 487 len 28672 data: 2 2 2 2 2 2 2 20/30 Read block 319 len 20480 21/30 Read block 448 len 36864 22/30 Read block 192 len 24576 23/30 Flush 24/30 Write block 11 len 32768 data: 5 4 4 4 4 4 2 2 25/30 Write block 155 len 8192 data: 3 3 26/30 Flush 27/30 Write block 44 len 32768 data: 4 4 4 4 3 3 3 3 28/30 Write block 300 len 16384 data: 3 3 3 3 29/30 Write block 73 len 8192 data: 2 3 30/30 Write block 241 len 16384 data: 3 3 3 3 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 2 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 8 Sent Sent Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 4 4 38 Done 0 26 26 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:24.357439443Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:24.357617648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1189,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 12 14 14 20 25 25 F F F <--- 1 020-039 12 12 12 20 20 20 F F F 2 040-059 12 12 12 20 20 20 F T T <--- 3 060-079 13 13 13 21 21 21 F T T <--- 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 11 14 14 17 25 25 F F F <--- 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 12 14 14 20 24 24 F F F <--- 11 220-239 12 14 14 19 24 24 F F F <--- 12 240-259 9 14 14 13 22 22 F T T <--- 13 260-279 13 14 14 21 22 22 F F F <--- 14 280-299 12 14 14 20 24 24 F F F <--- 15 300-319 8 14 14 12 24 24 F T T <--- 16 320-339 12 12 12 20 20 20 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 12 14 14 20 23 23 F F F <--- 21 420-439 10 10 10 15 15 15 F F F 22 440-459 10 14 14 16 24 24 F F F <--- 23 460-479 10 10 10 15 15 15 F F F 24 480-499 1 14 14 1 24 24 F F F <--- 25 500-519 9 14 14 13 23 23 F F F <--- 26 520-539 4 14 14 6 23 23 F F F <--- 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 10 14 14 14 24 24 F F F <--- Max gen: 14, Max flush: 25 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 12 12 12 20 20 20 F F F 2 040-059 15 15 15 26 26 26 F F F 3 060-079 15 15 15 26 26 26 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 14 14 14 25 25 25 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 14 14 14 24 24 24 F F F 11 220-239 14 14 14 24 24 24 F F F 12 240-259 15 15 15 26 26 26 F F F 13 260-279 14 14 14 22 22 22 F F F 14 280-299 14 14 14 24 24 24 F F F 15 300-319 15 15 15 26 26 26 F F F 16 320-339 12 12 12 20 20 20 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 14 14 14 23 23 23 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 14 14 14 24 24 24 F F F 23 460-479 10 10 10 15 15 15 F F F 24 480-499 14 14 14 24 24 24 F F F 25 500-519 14 14 14 23 23 23 F F F 26 520-539 14 14 14 23 23 23 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 14 14 14 24 24 24 F F F Max gen: 15, Max flush: 26 Begin loop 8 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.623768902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:28.630882962Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.630934991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.637668062Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.637949844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.637977873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.637986503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.637998302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Crucible d20b3bdd-6657-41ae-8b0b-578d79659b08 has session id: f50c2d28-e17d-4426-b57e-772806e27308","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638328543Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Upstairs opts: Upstairs UUID: d20b3bdd-6657-41ae-8b0b-578d79659b08, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638356822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Crucible stats registered with UUID: d20b3bdd-6657-41ae-8b0b-578d79659b08","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638367642Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:28.638378932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638524267Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638554877Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638565876Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638663744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638677863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"d20b3bdd-6657-41ae-8b0b-578d79659b08 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638714922Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638724572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638734322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638744051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638753551Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638763601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.63877347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"ds_connection connected from Ok(127.0.0.1:54742)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638822249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"ds_connection connected from Ok(127.0.0.1:47441)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638833289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"ds_connection connected from Ok(127.0.0.1:48800)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.638876218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639537719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639571038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639611137Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.63984045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.63987505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[0]R flush_numbers[0..12]: [25, 20, 26, 26, 3, 13, 21, 25, 10, 8, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639951747Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[0]R generation[0..12]: [14, 12, 15, 15, 3, 9, 13, 14, 6, 6, 14, 14]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639962997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639974087Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[1]R flush_numbers[0..12]: [25, 20, 26, 26, 3, 13, 21, 25, 10, 8, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639984576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[1]R generation[0..12]: [14, 12, 15, 15, 3, 9, 13, 14, 6, 6, 14, 14]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.639994746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640005896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[2]R flush_numbers[0..12]: [25, 20, 26, 26, 3, 13, 21, 25, 10, 8, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640016446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[2]R generation[0..12]: [14, 12, 15, 15, 3, 9, 13, 14, 6, 6, 14, 14]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640026735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640037675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640048435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640058044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Next flush: 27","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640067284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640076544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"":"downstairs","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640086584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640095393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"0","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640105423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"1","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640115213Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"client":"2","":"downstairs","session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640125232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640134542Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"d20b3bdd-6657-41ae-8b0b-578d79659b08 is now active with session: f50c2d28-e17d-4426-b57e-772806e27308","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640144422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640153062Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"session_id":"f50c2d28-e17d-4426-b57e-772806e27308"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640162021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Activated sub_volume d20b3bdd-6657-41ae-8b0b-578d79659b08","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640169921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640178041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.640489332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} 01/30 Write block 344 len 24576 data: 2 2 2 2 2 2 02/30 Write block 220 len 32768 data: 4 4 3 3 3 3 3 3 03/30 Read block 96 len 36864 04/30 Read block 92 len 4096 05/30 Read block 404 len 24576 06/30 Read block 40 len 4096 07/30 Read block 32 len 4096 08/30 Read block 518 len 20480 09/30 Flush 10/30 Write block 474 len 16384 data: 3 3 3 2 11/30 Read block 354 len 12288 12/30 Read block 386 len 4096 13/30 Write block 326 len 8192 data: 4 4 14/30 Read block 525 len 8192 15/30 Write block 564 len 16384 data: 3 3 3 3 16/30 Write block 65 len 40960 data: 3 5 4 4 4 4 2 2 3 4 17/30 Read block 275 len 8192 18/30 Write block 271 len 12288 data: 3 3 2 19/30 Write block 359 len 32768 data: 4 3 3 3 3 3 3 3 20/30 Read block 546 len 16384 21/30 Read block 118 len 36864 22/30 Flush 23/30 Write block 188 len 32768 data: 2 2 2 2 2 2 2 2 24/30 Read block 528 len 4096 25/30 Read block 441 len 40960 26/30 Write block 228 len 32768 data: 2 3 3 3 3 3 4 4 27/30 Read block 165 len 36864 28/30 Read block 493 len 16384 29/30 Write block 149 len 8192 data: 3 3 30/30 Write block 316 len 24576 data: 2 3 3 4 3 4 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:28.649253656Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:28.649428861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1197,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 12 12 12 20 20 20 F F F 2 040-059 15 15 15 26 26 26 F F F 3 060-079 15 16 16 26 28 28 F F F <--- 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 14 14 14 25 25 25 F T T <--- 8 160-179 6 6 6 10 10 10 F F F 9 180-199 6 6 6 8 8 8 F T T <--- 10 200-219 14 14 14 24 24 24 F F F 11 220-239 14 16 16 24 27 27 F T T <--- 12 240-259 15 15 15 26 26 26 F F F 13 260-279 14 16 16 22 28 28 F F F <--- 14 280-299 14 14 14 24 24 24 F F F 15 300-319 15 15 15 26 26 26 F T T <--- 16 320-339 12 16 16 20 28 28 F T T <--- 17 340-359 10 16 16 15 28 28 T F F <--- 18 360-379 10 16 16 15 28 28 F F F <--- 19 380-399 12 12 12 20 20 20 F F F 20 400-419 14 14 14 23 23 23 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 14 14 14 24 24 24 F F F 23 460-479 10 16 16 15 28 28 F F F <--- 24 480-499 14 14 14 24 24 24 F F F 25 500-519 14 14 14 23 23 23 F F F 26 520-539 14 14 14 23 23 23 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 10 16 16 16 28 28 F F F <--- 29 580-599 14 14 14 24 24 24 F F F Max gen: 16, Max flush: 28 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 12 12 12 20 20 20 F F F 2 040-059 15 15 15 26 26 26 F F F 3 060-079 16 16 16 28 28 28 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 17 17 17 29 29 29 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 17 17 17 29 29 29 F F F 10 200-219 14 14 14 24 24 24 F F F 11 220-239 17 17 17 29 29 29 F F F 12 240-259 15 15 15 26 26 26 F F F 13 260-279 16 16 16 28 28 28 F F F 14 280-299 14 14 14 24 24 24 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 17 17 17 29 29 29 F F F 17 340-359 16 16 16 28 28 28 F F F 18 360-379 16 16 16 28 28 28 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 14 14 14 23 23 23 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 14 14 14 24 24 24 F F F 23 460-479 16 16 16 28 28 28 F F F 24 480-499 14 14 14 24 24 24 F F F 25 500-519 14 14 14 23 23 23 F F F 26 520-539 14 14 14 23 23 23 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 14 14 14 24 24 24 F F F Max gen: 17, Max flush: 29 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.904633242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91289827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913187632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913216981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913225611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913237351Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Crucible d0e38a7d-0f54-4b55-8030-98819b165dea has session id: 399e540f-b8b1-496d-8e90-4f8861858bdd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913577671Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Upstairs opts: Upstairs UUID: d0e38a7d-0f54-4b55-8030-98819b165dea, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91360792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Crucible stats registered with UUID: d0e38a7d-0f54-4b55-8030-98819b165dea","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91361918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:32.91362833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913764986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913782745Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913792905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913927281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913940781Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"d0e38a7d-0f54-4b55-8030-98819b165dea active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91396798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91397877Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.913988509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914005539Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914015359Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914025428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914035498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:63605)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914112116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:34367)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914137415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:36153)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914197004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914805156Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914841945Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.914901554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915116438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915135957Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[0]R flush_numbers[0..12]: [25, 20, 26, 28, 3, 13, 21, 29, 10, 29, 24, 29]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915191446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[0]R generation[0..12]: [14, 12, 15, 16, 3, 9, 13, 17, 6, 17, 14, 17]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915202585Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915215905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[1]R flush_numbers[0..12]: [25, 20, 26, 28, 3, 13, 21, 29, 10, 29, 24, 29]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915226435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[1]R generation[0..12]: [14, 12, 15, 16, 3, 9, 13, 17, 6, 17, 14, 17]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915237074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915248024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[2]R flush_numbers[0..12]: [25, 20, 26, 28, 3, 13, 21, 29, 10, 29, 24, 29]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915258504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[2]R generation[0..12]: [14, 12, 15, 16, 3, 9, 13, 17, 6, 17, 14, 17]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915268823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915279803Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915289313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915299003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915308272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915317692Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"":"downstairs","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915327362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915336132Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"0","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915345791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"1","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915356631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"client":"2","":"downstairs","session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915366481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91537553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"d0e38a7d-0f54-4b55-8030-98819b165dea is now active with session: 399e540f-b8b1-496d-8e90-4f8861858bdd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.9153859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91539504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"session_id":"399e540f-b8b1-496d-8e90-4f8861858bdd"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.91540427Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Activated sub_volume d0e38a7d-0f54-4b55-8030-98819b165dea","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915412389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915420619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.915695961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} 01/30 Write block 369 len 4096 data: 3 02/30 Write block 292 len 28672 data: 2 2 2 2 2 2 3 03/30 Read block 240 len 20480 04/30 Read block 187 len 12288 05/30 Read block 30 len 24576 06/30 Write block 64 len 16384 data: 3 4 6 5 07/30 Flush 08/30 Write block 183 len 28672 data: 3 3 2 2 2 3 3 09/30 Read block 528 len 12288 10/30 Flush 11/30 Write block 239 len 12288 data: 3 3 4 12/30 Write block 209 len 16384 data: 4 3 3 4 13/30 Read block 67 len 20480 14/30 Write block 332 len 4096 data: 3 15/30 Flush 16/30 Write block 453 len 20480 data: 5 5 5 5 5 17/30 Write block 290 len 16384 data: 2 2 3 3 18/30 Write block 525 len 24576 data: 4 4 3 2 2 2 19/30 Write block 198 len 32768 data: 2 2 2 2 2 2 2 2 20/30 Write block 501 len 40960 data: 2 2 2 2 2 2 2 2 2 2 21/30 Write block 34 len 40960 data: 3 3 2 2 3 4 4 4 5 4 22/30 Read block 540 len 24576 23/30 Read block 346 len 8192 24/30 Write block 494 len 8192 data: 2 2 25/30 Write block 82 len 40960 data: 2 2 2 2 2 2 2 2 2 2 26/30 Write block 460 len 4096 data: 3 27/30 Write block 269 len 8192 data: 3 3 28/30 Read block 3 len 36864 29/30 Read block 372 len 12288 30/30 Write block 34 len 12288 data: 4 4 3 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 7 Done Done Done false 1008 Acked Read 3 Done Done Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 21 1 23 Done 22 2 22 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1006)) Some(JobId(1014)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:32.923349746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:32.92355915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1205,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 12 12 12 20 20 20 T F T <--- 2 040-059 15 15 15 26 26 26 T F T <--- 3 060-079 18 18 18 30 30 30 F F F 4 080-099 3 3 3 3 3 3 T F T <--- 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 17 17 17 29 29 29 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 18 17 18 31 29 31 T T T <--- 10 200-219 18 14 18 32 24 32 T F T <--- 11 220-239 18 17 18 32 29 32 F F F <--- 12 240-259 18 15 18 32 26 32 F F F <--- 13 260-279 16 16 16 28 28 28 T F T <--- 14 280-299 18 18 18 30 30 30 T F T <--- 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 17 18 32 29 32 F F F <--- 17 340-359 16 16 16 28 28 28 F F F 18 360-379 18 18 18 30 30 30 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 14 14 14 23 23 23 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 14 14 14 24 24 24 T F T <--- 23 460-479 16 16 16 28 28 28 T F T <--- 24 480-499 14 14 14 24 24 24 T F T <--- 25 500-519 14 14 14 23 23 23 T F T <--- 26 520-539 14 14 14 23 23 23 T F T <--- 27 540-559 10 10 10 15 15 15 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 14 14 14 24 24 24 F F F Max gen: 18, Max flush: 32 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 19 19 19 33 33 33 F F F 2 040-059 19 19 19 33 33 33 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 13 13 13 21 21 21 F F F 7 140-159 17 17 17 29 29 29 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 19 19 19 33 33 33 F F F 10 200-219 19 19 19 33 33 33 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 18 18 18 32 32 32 F F F 13 260-279 19 19 19 33 33 33 F F F 14 280-299 19 19 19 33 33 33 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 16 16 16 28 28 28 F F F 18 360-379 18 18 18 30 30 30 F F F 19 380-399 12 12 12 20 20 20 F F F 20 400-419 14 14 14 23 23 23 F F F 21 420-439 10 10 10 15 15 15 F F F 22 440-459 19 19 19 33 33 33 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 19 19 19 33 33 33 F F F 25 500-519 19 19 19 33 33 33 F F F 26 520-539 19 19 19 33 33 33 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 14 14 14 24 24 24 F F F Max gen: 19, Max flush: 33 Begin loop 10 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.182971034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191101566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191391598Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191422027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191430426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191442296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Crucible 8d04ccfe-5ea7-4ba1-9e85-30223dd5712e has session id: f26c2d84-f1e8-4e5d-8254-865f6a9f251b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191761327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Upstairs opts: Upstairs UUID: 8d04ccfe-5ea7-4ba1-9e85-30223dd5712e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191801596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Crucible stats registered with UUID: 8d04ccfe-5ea7-4ba1-9e85-30223dd5712e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191815166Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:37.191824365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.191987661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19201469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19202512Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192146786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192165706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"8d04ccfe-5ea7-4ba1-9e85-30223dd5712e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192196825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192206495Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192216284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192226274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192236124Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192246294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192256373Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"ds_connection connected from Ok(127.0.0.1:47948)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192320151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"ds_connection connected from Ok(127.0.0.1:49734)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192345921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"ds_connection connected from Ok(127.0.0.1:42870)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19235976Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.192972633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193000642Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193054291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193314323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193361262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[0]R flush_numbers[0..12]: [25, 33, 33, 30, 33, 13, 21, 29, 10, 33, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193403261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[0]R generation[0..12]: [14, 19, 19, 18, 19, 9, 13, 17, 6, 19, 19, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193414191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19342514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[1]R flush_numbers[0..12]: [25, 33, 33, 30, 33, 13, 21, 29, 10, 33, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19343553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[1]R generation[0..12]: [14, 19, 19, 18, 19, 9, 13, 17, 6, 19, 19, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.19344583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193456759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[2]R flush_numbers[0..12]: [25, 33, 33, 30, 33, 13, 21, 29, 10, 33, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193467309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[2]R generation[0..12]: [14, 19, 19, 18, 19, 9, 13, 17, 6, 19, 19, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193477889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193490879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193500288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193509968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Next flush: 34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193519278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193528797Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"":"downstairs","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193538997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193548067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"0","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193557977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"1","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193567796Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"client":"2","":"downstairs","session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193577906Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193587266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"8d04ccfe-5ea7-4ba1-9e85-30223dd5712e is now active with session: f26c2d84-f1e8-4e5d-8254-865f6a9f251b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193597206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193606295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193613965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"session_id":"f26c2d84-f1e8-4e5d-8254-865f6a9f251b"} {"msg":"Activated sub_volume 8d04ccfe-5ea7-4ba1-9e85-30223dd5712e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193623165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193631275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.193959945Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} 01/30 Write block 137 len 24576 data: 3 3 3 3 3 3 02/30 Read block 396 len 24576 03/30 Write block 588 len 20480 data: 6 4 3 3 2 04/30 Read block 108 len 24576 05/30 Read block 141 len 40960 06/30 Read block 50 len 40960 07/30 Write block 392 len 40960 data: 3 3 3 3 3 3 3 3 4 4 08/30 Read block 564 len 16384 09/30 Flush 10/30 Write block 48 len 40960 data: 4 4 4 4 3 3 2 2 2 3 11/30 Read block 331 len 12288 12/30 Flush 13/30 Write block 348 len 28672 data: 3 3 2 2 2 2 2 14/30 Read block 397 len 32768 15/30 Read block 33 len 24576 16/30 Read block 185 len 24576 17/30 Write block 513 len 36864 data: 3 3 2 2 2 3 3 3 3 18/30 Read block 570 len 40960 19/30 Write block 46 len 20480 data: 5 5 5 5 5 20/30 Write block 430 len 28672 data: 3 3 3 3 3 2 2 21/30 Read block 452 len 8192 22/30 Read block 135 len 20480 23/30 Write block 537 len 8192 data: 3 3 24/30 Read block 315 len 24576 25/30 Read block 550 len 12288 26/30 Write block 527 len 8192 data: 4 3 27/30 Read block 187 len 24576 28/30 Write block 203 len 4096 data: 3 29/30 Read block 172 len 8192 30/30 Read block 299 len 24576 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 6 Done Sent Done false 1002 Acked Write 5 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 0 30 Done 29 1 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) None Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:37.203513737Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:37.203681982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1213,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 19 19 19 33 33 33 F F F 2 040-059 20 19 20 35 33 35 T F T <--- 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 20 13 20 34 21 34 F T F <--- 7 140-159 20 17 20 34 29 34 F T F <--- 8 160-179 6 6 6 10 10 10 F F F 9 180-199 19 19 19 33 33 33 F F F 10 200-219 19 19 19 33 33 33 T F T <--- 11 220-239 18 18 18 32 32 32 F F F 12 240-259 18 18 18 32 32 32 F F F 13 260-279 19 19 19 33 33 33 F F F 14 280-299 19 19 19 33 33 33 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 16 16 16 28 28 28 T F T <--- 18 360-379 18 18 18 30 30 30 F F F 19 380-399 20 12 20 34 20 34 F F F <--- 20 400-419 20 14 20 34 23 34 F F F <--- 21 420-439 10 10 10 15 15 15 T F T <--- 22 440-459 19 19 19 33 33 33 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 19 19 19 33 33 33 F F F 25 500-519 19 19 19 33 33 33 T F T <--- 26 520-539 19 19 19 33 33 33 T F T <--- 27 540-559 10 10 10 15 15 15 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 20 14 20 34 24 34 F F F <--- Max gen: 20, Max flush: 35 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 19 19 19 33 33 33 F F F 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 20 20 20 34 34 34 F F F 7 140-159 20 20 20 34 34 34 F F F 8 160-179 6 6 6 10 10 10 F F F 9 180-199 19 19 19 33 33 33 F F F 10 200-219 21 21 21 36 36 36 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 18 18 18 32 32 32 F F F 13 260-279 19 19 19 33 33 33 F F F 14 280-299 19 19 19 33 33 33 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 21 21 21 36 36 36 F F F 18 360-379 18 18 18 30 30 30 F F F 19 380-399 20 20 20 34 34 34 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 21 21 21 36 36 36 F F F 22 440-459 19 19 19 33 33 33 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 19 19 19 33 33 33 F F F 25 500-519 21 21 21 36 36 36 F F F 26 520-539 21 21 21 36 36 36 F F F 27 540-559 10 10 10 15 15 15 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 20 20 20 34 34 34 F F F Max gen: 21, Max flush: 36 Begin loop 11 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.4551193Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.463354389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.46364573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.46367541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.463683679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.463695229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Crucible b97898dd-904a-4d78-881c-3d2c846d75b5 has session id: a32da661-09df-4ff0-8ca7-e2dcf546916b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.46401067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Upstairs opts: Upstairs UUID: b97898dd-904a-4d78-881c-3d2c846d75b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464044399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Crucible stats registered with UUID: b97898dd-904a-4d78-881c-3d2c846d75b5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464055199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:41.464064229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464207035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464223214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464233234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.46436151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.46438665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"b97898dd-904a-4d78-881c-3d2c846d75b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464395609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464404859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464415049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464424828Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464434478Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464444488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464454498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"ds_connection connected from Ok(127.0.0.1:58006)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464501796Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"ds_connection connected from Ok(127.0.0.1:39046)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464527416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"ds_connection connected from Ok(127.0.0.1:44030)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.464538795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465101949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465120149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465131169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.4654219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.4654444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[0]R flush_numbers[0..12]: [25, 33, 36, 30, 33, 13, 34, 34, 10, 33, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465457289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[0]R generation[0..12]: [14, 19, 21, 18, 19, 9, 20, 20, 6, 19, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465468599Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465479759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[1]R flush_numbers[0..12]: [25, 33, 36, 30, 33, 13, 34, 34, 10, 33, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465490439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[1]R generation[0..12]: [14, 19, 21, 18, 19, 9, 20, 20, 6, 19, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465500788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465511748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[2]R flush_numbers[0..12]: [25, 33, 36, 30, 33, 13, 34, 34, 10, 33, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465522388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[2]R generation[0..12]: [14, 19, 21, 18, 19, 9, 20, 20, 6, 19, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465532757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465543717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465553247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465562916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465572416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465581976Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"":"downstairs","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465591866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465602505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"0","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465612455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"1","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465622235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"client":"2","":"downstairs","session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465632275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465641474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"b97898dd-904a-4d78-881c-3d2c846d75b5 is now active with session: a32da661-09df-4ff0-8ca7-e2dcf546916b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465651694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465660604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"session_id":"a32da661-09df-4ff0-8ca7-e2dcf546916b"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465669833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Activated sub_volume b97898dd-904a-4d78-881c-3d2c846d75b5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465677893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.465686383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.466027743Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} 01/30 Flush 02/30 Write block 274 len 8192 data: 2 4 03/30 Write block 425 len 28672 data: 3 3 3 3 3 4 4 04/30 Write block 296 len 12288 data: 3 3 4 05/30 Write block 488 len 24576 data: 3 3 3 3 3 3 06/30 Read block 518 len 28672 07/30 Read block 383 len 24576 08/30 Write block 538 len 28672 data: 4 3 4 4 5 5 4 09/30 Read block 277 len 8192 10/30 Write block 162 len 28672 data: 3 3 3 2 2 2 2 11/30 Read block 24 len 24576 12/30 Read block 372 len 28672 13/30 Read block 146 len 36864 14/30 Read block 149 len 20480 15/30 Flush 16/30 Write block 531 len 16384 data: 2 2 2 2 17/30 Read block 324 len 24576 18/30 Write block 28 len 4096 data: 3 19/30 Read block 143 len 16384 20/30 Flush 21/30 Write block 187 len 36864 data: 3 4 4 3 3 3 3 3 3 22/30 Write block 346 len 8192 data: 3 3 23/30 Flush 24/30 Write block 249 len 36864 data: 3 3 3 3 3 3 2 2 2 25/30 Write block 133 len 36864 data: 2 2 2 3 4 4 4 4 4 26/30 Write block 188 len 20480 data: 5 5 4 4 4 27/30 Write block 533 len 8192 data: 3 3 28/30 Read block 96 len 20480 29/30 Read block 432 len 12288 30/30 Write block 549 len 32768 data: 2 2 2 3 3 4 4 4 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 7 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:41.474451627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:41.474609982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1221,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 22 22 19 39 39 33 F F F <--- 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 20 20 20 34 34 34 T T F <--- 7 140-159 20 20 20 34 34 34 T T F <--- 8 160-179 22 22 6 38 38 10 F F F <--- 9 180-199 22 22 19 40 40 33 T T F <--- 10 200-219 21 21 21 36 36 36 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 18 18 18 32 32 32 T T F <--- 13 260-279 22 22 19 38 38 33 F F F <--- 14 280-299 22 22 19 38 38 33 F F F <--- 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 22 22 21 40 40 36 F F F <--- 18 360-379 18 18 18 30 30 30 F F F 19 380-399 20 20 20 34 34 34 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 21 38 38 36 F F F <--- 22 440-459 19 19 19 33 33 33 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 22 22 19 38 38 33 F F F <--- 25 500-519 21 21 21 36 36 36 F F F 26 520-539 22 22 21 39 39 36 T T F <--- 27 540-559 22 22 10 38 38 15 T T F <--- 28 560-579 16 16 16 28 28 28 F F F 29 580-599 20 20 20 34 34 34 F F F Max gen: 22, Max flush: 40 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 F F F 1 020-039 22 22 22 39 39 39 F F F 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 9 9 9 13 13 13 F F F 6 120-139 23 23 23 41 41 41 F F F 7 140-159 23 23 23 41 41 41 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 23 23 23 41 41 41 F F F 10 200-219 21 21 21 36 36 36 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 22 22 22 38 38 38 F F F 14 280-299 22 22 22 38 38 38 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 22 22 22 40 40 40 F F F 18 360-379 18 18 18 30 30 30 F F F 19 380-399 20 20 20 34 34 34 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 22 38 38 38 F F F 22 440-459 19 19 19 33 33 33 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 22 22 22 38 38 38 F F F 25 500-519 21 21 21 36 36 36 F F F 26 520-539 23 23 23 41 41 41 F F F 27 540-559 23 23 23 41 41 41 F F F 28 560-579 16 16 16 28 28 28 F F F 29 580-599 20 20 20 34 34 34 F F F Max gen: 23, Max flush: 41 Begin loop 12 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.728636737Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.736806258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737104549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737133359Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737141988Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737153758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Crucible 78c0ea04-d9d1-4ea0-98c4-20d9a9277bac has session id: 89d88e8f-41bc-4ba0-9de3-5038cd17e43f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737489179Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Upstairs opts: Upstairs UUID: 78c0ea04-d9d1-4ea0-98c4-20d9a9277bac, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737519568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Crucible stats registered with UUID: 78c0ea04-d9d1-4ea0-98c4-20d9a9277bac","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737530787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:45.737540097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737683243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737714822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737725822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737841399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737859078Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"78c0ea04-d9d1-4ea0-98c4-20d9a9277bac active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737886447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737895787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737905497Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737915217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737924826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737934696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.737946746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35223)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.738008004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"ds_connection connected from Ok(127.0.0.1:53246)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.738092402Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"ds_connection connected from Ok(127.0.0.1:39357)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.738117291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.738750163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.738766863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.73887608Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739102423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739112473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[0]R flush_numbers[0..12]: [25, 39, 36, 30, 33, 13, 41, 41, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739182271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[0]R generation[0..12]: [14, 22, 21, 18, 19, 9, 23, 23, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739193361Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.73920451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[1]R flush_numbers[0..12]: [25, 39, 36, 30, 33, 13, 41, 41, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.73921525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[1]R generation[0..12]: [14, 22, 21, 18, 19, 9, 23, 23, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.73922594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739237209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[2]R flush_numbers[0..12]: [25, 39, 36, 30, 33, 13, 41, 41, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739249559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[2]R generation[0..12]: [14, 22, 21, 18, 19, 9, 23, 23, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739260159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739271378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739280948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739290708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Next flush: 42","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739300118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739309687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"":"downstairs","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739319487Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739328417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"0","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739338097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"1","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739347836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"client":"2","":"downstairs","session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739357856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739366956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"78c0ea04-d9d1-4ea0-98c4-20d9a9277bac is now active with session: 89d88e8f-41bc-4ba0-9de3-5038cd17e43f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739377035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739385815Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"session_id":"89d88e8f-41bc-4ba0-9de3-5038cd17e43f"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739396345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Activated sub_volume 78c0ea04-d9d1-4ea0-98c4-20d9a9277bac","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739404465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739412604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.739722726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} 01/30 Write block 116 len 12288 data: 4 4 3 02/30 Read block 67 len 4096 03/30 Read block 483 len 24576 04/30 Flush 05/30 Write block 335 len 8192 data: 3 3 06/30 Write block 104 len 24576 data: 2 2 2 2 2 3 07/30 Read block 537 len 12288 08/30 Read block 551 len 8192 09/30 Read block 101 len 40960 10/30 Write block 561 len 32768 data: 3 3 3 4 4 4 4 4 11/30 Read block 22 len 8192 12/30 Flush 13/30 Write block 118 len 40960 data: 4 3 2 2 2 3 3 3 3 4 14/30 Read block 575 len 32768 15/30 Flush 16/30 Write block 377 len 36864 data: 4 3 3 3 4 4 5 6 6 17/30 Write block 456 len 16384 data: 6 6 4 3 18/30 Write block 519 len 36864 data: 4 4 4 3 4 4 5 5 5 19/30 Flush 20/30 Write block 145 len 8192 data: 3 2 21/30 Flush 22/30 Write block 349 len 32768 data: 4 3 3 3 3 3 2 2 23/30 Write block 492 len 32768 data: 4 4 3 3 2 2 2 2 24/30 Read block 178 len 28672 25/30 Read block 50 len 32768 26/30 Write block 2 len 12288 data: 2 2 2 27/30 Write block 543 len 8192 data: 6 5 28/30 Read block 461 len 16384 29/30 Read block 308 len 20480 30/30 Read block 62 len 28672 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 1 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:45.748940167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:45.749117852Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1229,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 14 14 14 25 25 25 T F T <--- 1 020-039 22 22 22 39 39 39 F F F 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 24 9 24 44 13 44 F T F <--- 6 120-139 24 23 24 44 41 44 F F F <--- 7 140-159 24 23 24 46 41 46 F F F <--- 8 160-179 22 22 22 38 38 38 F F F 9 180-199 23 23 23 41 41 41 F F F 10 200-219 21 21 21 36 36 36 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 22 22 22 38 38 38 F F F 14 280-299 22 22 22 38 38 38 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 18 24 43 32 43 F F F <--- 17 340-359 22 22 22 40 40 40 T F T <--- 18 360-379 24 18 24 45 30 45 F F F <--- 19 380-399 24 20 24 45 34 45 F F F <--- 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 22 38 38 38 F F F 22 440-459 24 19 24 45 33 45 F F F <--- 23 460-479 19 19 19 33 33 33 F F F 24 480-499 22 22 22 38 38 38 T F T <--- 25 500-519 24 21 24 45 36 45 F F F <--- 26 520-539 24 23 24 45 41 45 F F F <--- 27 540-559 23 23 23 41 41 41 T F T <--- 28 560-579 24 16 24 43 28 43 F F F <--- 29 580-599 20 20 20 34 34 34 F F F Max gen: 24, Max flush: 46 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 47 47 47 F F F 1 020-039 22 22 22 39 39 39 F F F 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 18 18 30 30 30 F F F 4 080-099 19 19 19 33 33 33 F F F 5 100-119 24 24 24 44 44 44 F F F 6 120-139 24 24 24 44 44 44 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 23 23 23 41 41 41 F F F 10 200-219 21 21 21 36 36 36 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 22 22 22 38 38 38 F F F 14 280-299 22 22 22 38 38 38 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 22 38 38 38 F F F 22 440-459 24 24 24 45 45 45 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 25 25 25 47 47 47 F F F 25 500-519 24 24 24 45 45 45 F F F 26 520-539 24 24 24 45 45 45 F F F 27 540-559 25 25 25 47 47 47 F F F 28 560-579 24 24 24 43 43 43 F F F 29 580-599 20 20 20 34 34 34 F F F Max gen: 25, Max flush: 47 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.003079938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.011628518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.01191888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.011948989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.011957498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.011969188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Crucible fbc5edd0-6cc0-463b-8306-63c69e755807 has session id: 6cf25bf3-5064-4d5e-b009-b436f7f86c41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012339118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Upstairs opts: Upstairs UUID: fbc5edd0-6cc0-463b-8306-63c69e755807, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012371887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Crucible stats registered with UUID: fbc5edd0-6cc0-463b-8306-63c69e755807","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012382657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:50.012391746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012549632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012581391Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012592481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012708857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012719457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"fbc5edd0-6cc0-463b-8306-63c69e755807 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012743006Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012752326Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012762126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012771826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012781415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012791315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012801295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"ds_connection connected from Ok(127.0.0.1:45264)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012880123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"ds_connection connected from Ok(127.0.0.1:53773)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012906262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"ds_connection connected from Ok(127.0.0.1:41681)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.012917732Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.013586313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.013646121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.013660441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.013924893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.013971392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[0]R flush_numbers[0..12]: [47, 39, 36, 30, 33, 44, 44, 46, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014010491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[0]R generation[0..12]: [25, 22, 21, 18, 19, 24, 24, 24, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.01402186Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.01403364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[1]R flush_numbers[0..12]: [47, 39, 36, 30, 33, 44, 44, 46, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.01404437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[1]R generation[0..12]: [25, 22, 21, 18, 19, 24, 24, 24, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.0140549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014066579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[2]R flush_numbers[0..12]: [47, 39, 36, 30, 33, 44, 44, 46, 38, 41, 36, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014077239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[2]R generation[0..12]: [25, 22, 21, 18, 19, 24, 24, 24, 22, 23, 21, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014087659Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014098648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014108048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014117688Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Next flush: 48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014128997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014138517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"":"downstairs","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014148317Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014157407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"0","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014167356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"1","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014177146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"client":"2","":"downstairs","session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014187196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014196476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"fbc5edd0-6cc0-463b-8306-63c69e755807 is now active with session: 6cf25bf3-5064-4d5e-b009-b436f7f86c41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014206635Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014215505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"session_id":"6cf25bf3-5064-4d5e-b009-b436f7f86c41"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014224505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Activated sub_volume fbc5edd0-6cc0-463b-8306-63c69e755807","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014232705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014240804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.014600614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} 01/30 Write block 592 len 12288 data: 3 2 2 02/30 Read block 120 len 40960 03/30 Flush 04/30 Write block 482 len 8192 data: 2 2 05/30 Write block 444 len 4096 data: 3 06/30 Read block 224 len 16384 07/30 Read block 31 len 32768 08/30 Read block 404 len 32768 09/30 Read block 296 len 32768 10/30 Read block 588 len 8192 11/30 Read block 209 len 12288 12/30 Write block 75 len 28672 data: 3 3 3 3 3 3 2 13/30 Read block 529 len 36864 14/30 Write block 197 len 36864 data: 2 3 3 3 3 3 4 3 3 15/30 Read block 455 len 24576 16/30 Flush 17/30 Write block 592 len 12288 data: 4 3 3 18/30 Flush 19/30 Write block 28 len 36864 data: 4 3 3 3 3 3 5 5 4 20/30 Read block 421 len 36864 21/30 Read block 69 len 12288 22/30 Read block 563 len 32768 23/30 Read block 96 len 36864 24/30 Read block 375 len 28672 25/30 Read block 123 len 8192 26/30 Read block 212 len 36864 27/30 Read block 49 len 36864 28/30 Write block 451 len 8192 data: 4 4 29/30 Write block 276 len 20480 data: 2 3 3 3 2 30/30 Write block 480 len 8192 data: 2 2 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 2 Sent Sent Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 3 2 33 Done 2 27 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:50.023905293Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:50.024084668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1237,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 47 47 47 F F F 1 020-039 22 22 22 39 39 39 F T T <--- 2 040-059 21 21 21 36 36 36 F F F 3 060-079 18 26 26 30 49 49 F F F <--- 4 080-099 19 26 26 33 49 49 F F F <--- 5 100-119 24 24 24 44 44 44 F F F 6 120-139 24 24 24 44 44 44 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 23 26 26 41 49 49 F F F <--- 10 200-219 21 26 26 36 49 49 F F F <--- 11 220-239 18 18 18 32 32 32 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 22 22 22 38 38 38 F T T <--- 14 280-299 22 22 22 38 38 38 F T T <--- 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 22 38 38 38 F F F 22 440-459 24 26 26 45 49 49 F T T <--- 23 460-479 19 19 19 33 33 33 F F F 24 480-499 25 26 26 47 49 49 F T T <--- 25 500-519 24 24 24 45 45 45 F F F 26 520-539 24 24 24 45 45 45 F F F 27 540-559 25 25 25 47 47 47 F F F 28 560-579 24 24 24 43 43 43 F F F 29 580-599 20 26 26 34 50 50 T F F <--- Max gen: 26, Max flush: 50 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 47 47 47 F F F 1 020-039 27 27 27 51 51 51 F F F 2 040-059 21 21 21 36 36 36 F F F 3 060-079 26 26 26 49 49 49 F F F 4 080-099 26 26 26 49 49 49 F F F 5 100-119 24 24 24 44 44 44 F F F 6 120-139 24 24 24 44 44 44 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 26 26 26 49 49 49 F F F 10 200-219 26 26 26 49 49 49 F F F 11 220-239 18 18 18 32 32 32 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 27 27 27 51 51 51 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 22 22 22 38 38 38 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 27 27 27 51 51 51 F F F 25 500-519 24 24 24 45 45 45 F F F 26 520-539 24 24 24 45 45 45 F F F 27 540-559 25 25 25 47 47 47 F F F 28 560-579 24 24 24 43 43 43 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 27, Max flush: 51 Begin loop 14 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.281226185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.289500783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.289795234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.289823274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.289834483Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.289846263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Crucible 33b5ef9e-6724-4609-85c1-ba5b337afccc has session id: 293cd984-6718-499e-a102-298ee7f0a88b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290265891Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Upstairs opts: Upstairs UUID: 33b5ef9e-6724-4609-85c1-ba5b337afccc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29029479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Crucible stats registered with UUID: 33b5ef9e-6724-4609-85c1-ba5b337afccc","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29030625Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:54.2903155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290468005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290485175Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290495245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290629611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290643811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"33b5ef9e-6724-4609-85c1-ba5b337afccc active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290681699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290691059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290700799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290710579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290722768Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290732848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290742858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"ds_connection connected from Ok(127.0.0.1:58003)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290791226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"ds_connection connected from Ok(127.0.0.1:65336)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290832895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"ds_connection connected from Ok(127.0.0.1:63052)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.290849675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291503116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291516706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291538345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291805308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291829947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[0]R flush_numbers[0..12]: [47, 51, 36, 49, 49, 44, 44, 46, 38, 49, 49, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291881676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[0]R generation[0..12]: [25, 27, 21, 26, 26, 24, 24, 24, 22, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291893325Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291904525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[1]R flush_numbers[0..12]: [47, 51, 36, 49, 49, 44, 44, 46, 38, 49, 49, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291915485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[1]R generation[0..12]: [25, 27, 21, 26, 26, 24, 24, 24, 22, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291929284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291940544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[2]R flush_numbers[0..12]: [47, 51, 36, 49, 49, 44, 44, 46, 38, 49, 49, 32]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291951174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[2]R generation[0..12]: [25, 27, 21, 26, 26, 24, 24, 24, 22, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291961743Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291972703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291982083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.291991893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Next flush: 52","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292012672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292022142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"":"downstairs","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292032032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292040731Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"0","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292050511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"1","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292060291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"client":"2","":"downstairs","session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29207034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29207982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"33b5ef9e-6724-4609-85c1-ba5b337afccc is now active with session: 293cd984-6718-499e-a102-298ee7f0a88b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29209162Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29210044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"session_id":"293cd984-6718-499e-a102-298ee7f0a88b"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292109749Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Activated sub_volume 33b5ef9e-6724-4609-85c1-ba5b337afccc","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292118099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.292126519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.29245818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} 01/30 Write block 47 len 16384 data: 6 6 6 6 02/30 Write block 428 len 16384 data: 4 4 5 5 03/30 Read block 408 len 20480 04/30 Read block 113 len 4096 05/30 Write block 95 len 40960 data: 2 2 2 2 2 2 2 2 2 3 06/30 Read block 109 len 24576 07/30 Write block 439 len 4096 data: 2 08/30 Read block 218 len 12288 09/30 Read block 103 len 36864 10/30 Flush 11/30 Write block 221 len 40960 data: 5 4 4 4 4 4 4 3 4 4 12/30 Flush 13/30 Flush 14/30 Write block 69 len 12288 data: 5 5 3 15/30 Write block 510 len 32768 data: 3 3 3 4 4 3 3 3 16/30 Read block 569 len 28672 17/30 Read block 234 len 4096 18/30 Read block 65 len 28672 19/30 Read block 376 len 8192 20/30 Read block 584 len 20480 21/30 Write block 207 len 28672 data: 2 2 5 4 4 5 3 22/30 Write block 4 len 20480 data: 3 2 3 4 4 23/30 Read block 346 len 12288 24/30 Read block 408 len 32768 25/30 Write block 119 len 40960 data: 4 3 3 3 4 4 4 4 5 4 26/30 Write block 83 len 8192 data: 3 3 27/30 Read block 273 len 32768 28/30 Write block 491 len 4096 data: 4 29/30 Read block 44 len 28672 30/30 Read block 19 len 4096 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 1 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Write 1 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Read 9 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 22 0 22 Done 30 8 30 68 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:54.301396548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:54.301567623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1245,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 47 47 47 T F T <--- 1 020-039 27 27 27 51 51 51 F F F 2 040-059 28 21 28 52 36 52 F T F <--- 3 060-079 26 26 26 49 49 49 T F T <--- 4 080-099 28 26 28 52 49 52 T T T <--- 5 100-119 28 24 28 52 44 52 T T T <--- 6 120-139 24 24 24 44 44 44 T F T <--- 7 140-159 24 24 24 46 46 46 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 26 26 26 49 49 49 F F F 10 200-219 26 26 26 49 49 49 T F T <--- 11 220-239 28 18 28 53 32 53 F F F <--- 12 240-259 23 23 23 41 41 41 F F F 13 260-279 27 27 27 51 51 51 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 28 22 28 52 38 52 F T F <--- 22 440-459 27 27 27 51 51 51 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 27 27 27 51 51 51 T F T <--- 25 500-519 24 24 24 45 45 45 T F T <--- 26 520-539 24 24 24 45 45 45 F F F 27 540-559 25 25 25 47 47 47 F F F 28 560-579 24 24 24 43 43 43 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 28, Max flush: 53 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 29 29 29 54 54 54 F F F 1 020-039 27 27 27 51 51 51 F F F 2 040-059 28 28 28 52 52 52 F F F 3 060-079 29 29 29 54 54 54 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 22 22 22 38 38 38 F F F 9 180-199 26 26 26 49 49 49 F F F 10 200-219 29 29 29 54 54 54 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 23 23 23 41 41 41 F F F 13 260-279 27 27 27 51 51 51 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 F F F 20 400-419 20 20 20 34 34 34 F F F 21 420-439 28 28 28 52 52 52 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 29 29 29 54 54 54 F F F 25 500-519 29 29 29 54 54 54 F F F 26 520-539 24 24 24 45 45 45 F F F 27 540-559 25 25 25 47 47 47 F F F 28 560-579 24 24 24 43 43 43 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 29, Max flush: 54 Begin loop 15 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.558991383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567180123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567470395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567495834Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567504614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567516224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Crucible fd046c4b-9428-4a5a-be20-afae80de3787 has session id: a93f99f3-16e2-4975-b96d-52eec926be6c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567866294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Upstairs opts: Upstairs UUID: fd046c4b-9428-4a5a-be20-afae80de3787, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567913272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Crucible stats registered with UUID: fd046c4b-9428-4a5a-be20-afae80de3787","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.567923522Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:58.567934992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568072198Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568088687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568098757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568228964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568243063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"fd046c4b-9428-4a5a-be20-afae80de3787 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568279032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568288422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568298182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568307821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568317251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568327231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.56833715Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"ds_connection connected from Ok(127.0.0.1:62879)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.56836366Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45145)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568374559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"ds_connection connected from Ok(127.0.0.1:64124)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568391559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568961653Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.568990722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569101589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569246035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569255745Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[0]R flush_numbers[0..12]: [54, 51, 52, 54, 54, 54, 54, 46, 38, 49, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569389551Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[0]R generation[0..12]: [29, 27, 28, 29, 29, 29, 29, 24, 22, 26, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569400701Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.56941183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[1]R flush_numbers[0..12]: [54, 51, 52, 54, 54, 54, 54, 46, 38, 49, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.56942255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[1]R generation[0..12]: [29, 27, 28, 29, 29, 29, 29, 24, 22, 26, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.56943295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569444099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[2]R flush_numbers[0..12]: [54, 51, 52, 54, 54, 54, 54, 46, 38, 49, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569454549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[2]R generation[0..12]: [29, 27, 28, 29, 29, 29, 29, 24, 22, 26, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569465159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569476058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569487758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569497518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Next flush: 55","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569506888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569516237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"":"downstairs","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569525937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569534687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"0","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569544477Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"1","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569554266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"client":"2","":"downstairs","session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569564146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569573286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"fd046c4b-9428-4a5a-be20-afae80de3787 is now active with session: a93f99f3-16e2-4975-b96d-52eec926be6c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569583285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569592155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"session_id":"a93f99f3-16e2-4975-b96d-52eec926be6c"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569601245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Activated sub_volume fd046c4b-9428-4a5a-be20-afae80de3787","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569609555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-09-24T22:09:58.569617764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.569900217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} 01/30 Write block 400 len 32768 data: 5 5 4 3 3 3 3 3 02/30 Read block 44 len 40960 03/30 Write block 256 len 28672 data: 3 3 2 3 3 3 3 04/30 Read block 513 len 16384 05/30 Write block 37 len 36864 data: 3 4 5 5 5 6 5 5 5 06/30 Write block 449 len 40960 data: 3 4 5 5 6 6 6 7 7 5 07/30 Write block 557 len 24576 data: 4 3 3 4 4 4 08/30 Read block 488 len 32768 09/30 Flush 10/30 Write block 435 len 28672 data: 3 3 2 2 3 2 2 11/30 Write block 168 len 36864 data: 3 2 2 2 2 2 2 2 2 12/30 Write block 514 len 32768 data: 5 4 4 4 4 5 5 5 13/30 Write block 184 len 32768 data: 4 3 3 4 6 6 5 5 14/30 Write block 22 len 20480 data: 4 4 4 3 3 15/30 Read block 420 len 16384 16/30 Write block 79 len 4096 data: 4 17/30 Write block 454 len 4096 data: 7 18/30 Read block 418 len 40960 19/30 Flush 20/30 Write block 555 len 32768 data: 5 5 5 4 4 5 5 5 21/30 Read block 114 len 12288 22/30 Read block 150 len 32768 23/30 Read block 293 len 8192 24/30 Write block 570 len 8192 data: 3 2 25/30 Read block 249 len 40960 26/30 Read block 319 len 40960 27/30 Write block 535 len 12288 data: 3 3 4 28/30 Write block 536 len 36864 data: 4 5 5 4 5 5 6 7 6 29/30 Read block 10 len 20480 30/30 Write block 385 len 36864 data: 7 6 7 7 7 5 3 4 4 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 7 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 29 29 2 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:09:58.578591772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:09:58.578776167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1253,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 29 29 29 54 54 54 F F F 1 020-039 30 30 27 56 56 51 F F F <--- 2 040-059 30 30 28 55 55 52 F F F <--- 3 060-079 30 30 29 56 56 54 F F F <--- 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 30 30 22 56 56 38 F F F <--- 9 180-199 30 30 26 56 56 49 F F F <--- 10 200-219 29 29 29 54 54 54 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 30 30 23 55 55 41 F F F <--- 13 260-279 30 30 27 55 55 51 F F F <--- 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 24 24 24 45 45 45 T T F <--- 20 400-419 30 30 20 55 55 34 F F T <--- 21 420-439 30 30 28 56 56 52 F F F <--- 22 440-459 30 30 27 56 56 51 F F F <--- 23 460-479 19 19 19 33 33 33 F F F 24 480-499 29 29 29 54 54 54 F F F 25 500-519 30 30 29 56 56 54 F F F <--- 26 520-539 30 30 24 56 56 45 T T F <--- 27 540-559 30 30 25 55 55 47 T T F <--- 28 560-579 30 30 24 55 55 43 T T F <--- 29 580-599 26 26 26 50 50 50 F F F Max gen: 30, Max flush: 56 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 29 29 29 54 54 54 F F F 1 020-039 30 30 30 56 56 56 F F F 2 040-059 30 30 30 55 55 55 F F F 3 060-079 30 30 30 56 56 56 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 30 30 30 56 56 56 F F F 9 180-199 30 30 30 56 56 56 F F F 10 200-219 29 29 29 54 54 54 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 30 30 30 55 55 55 F F F 13 260-279 30 30 30 55 55 55 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 24 24 24 43 43 43 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 30 30 30 56 56 56 F F F 22 440-459 30 30 30 56 56 56 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 29 29 29 54 54 54 F F F 25 500-519 30 30 30 56 56 56 F F F 26 520-539 31 31 31 57 57 57 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 31, Max flush: 57 Begin loop 16 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.839449315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.847683163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.847986575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848016784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848025914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848037573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Crucible 66eb878e-5cd9-4796-b26d-2a75a59ab42c has session id: c93df73c-ba14-443e-b6f4-b7ef9dc7ac48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848395543Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Upstairs opts: Upstairs UUID: 66eb878e-5cd9-4796-b26d-2a75a59ab42c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848425302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Crucible stats registered with UUID: 66eb878e-5cd9-4796-b26d-2a75a59ab42c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848482281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:02.848491711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848588728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848605437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848615627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848759793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848778073Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"66eb878e-5cd9-4796-b26d-2a75a59ab42c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848804452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848813832Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848823701Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848833441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848843211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.84885328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.84886347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"ds_connection connected from Ok(127.0.0.1:46551)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848904719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"ds_connection connected from Ok(127.0.0.1:34175)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848915819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"ds_connection connected from Ok(127.0.0.1:58641)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.848926128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.849646048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.849691857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.849705536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.849979959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850025857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[0]R flush_numbers[0..12]: [54, 56, 55, 56, 54, 54, 54, 46, 56, 56, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850072896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[0]R generation[0..12]: [29, 30, 30, 30, 29, 29, 29, 24, 30, 30, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850084276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850095656Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[1]R flush_numbers[0..12]: [54, 56, 55, 56, 54, 54, 54, 46, 56, 56, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850106175Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[1]R generation[0..12]: [29, 30, 30, 30, 29, 29, 29, 24, 30, 30, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850116655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850127695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[2]R flush_numbers[0..12]: [54, 56, 55, 56, 54, 54, 54, 46, 56, 56, 54, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850138214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[2]R generation[0..12]: [29, 30, 30, 30, 29, 29, 29, 24, 30, 30, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850148634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850159764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850169223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850179183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Next flush: 58","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850188553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850198093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"":"downstairs","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850207892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850216812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"0","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850228082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"1","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850237882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"client":"2","":"downstairs","session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850247831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850256941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"66eb878e-5cd9-4796-b26d-2a75a59ab42c is now active with session: c93df73c-ba14-443e-b6f4-b7ef9dc7ac48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850266941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.85027569Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"session_id":"c93df73c-ba14-443e-b6f4-b7ef9dc7ac48"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.85028522Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Activated sub_volume 66eb878e-5cd9-4796-b26d-2a75a59ab42c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.85029328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.85030142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.850603471Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} 01/30 Flush 02/30 Write block 18 len 24576 data: 3 4 5 4 5 5 03/30 Read block 274 len 12288 04/30 Write block 37 len 32768 data: 4 5 6 6 6 7 6 6 05/30 Read block 590 len 36864 06/30 Read block 278 len 4096 07/30 Write block 65 len 16384 data: 5 7 6 5 08/30 Write block 249 len 28672 data: 4 4 4 4 4 4 3 09/30 Write block 328 len 40960 data: 4 6 6 4 4 3 3 4 4 3 10/30 Read block 5 len 20480 11/30 Flush 12/30 Write block 19 len 24576 data: 5 6 5 6 6 5 13/30 Read block 529 len 36864 14/30 Read block 346 len 8192 15/30 Read block 569 len 12288 16/30 Write block 482 len 16384 data: 3 3 2 2 17/30 Read block 241 len 16384 18/30 Read block 454 len 4096 19/30 Read block 587 len 20480 20/30 Write block 215 len 16384 data: 3 3 3 3 21/30 Write block 264 len 40960 data: 3 4 3 3 3 4 4 4 4 3 22/30 Read block 10 len 8192 23/30 Read block 278 len 8192 24/30 Read block 399 len 40960 25/30 Read block 424 len 40960 26/30 Read block 130 len 28672 27/30 Write block 377 len 4096 data: 5 28/30 Read block 529 len 28672 29/30 Write block 518 len 28672 data: 5 6 6 6 4 5 5 30/30 Write block 444 len 12288 data: 4 4 4 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Done Sent false 1001 Acked Write 6 Done Done Sent false 1002 Acked Read 3 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Write 7 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:02.859479982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:02.859652847Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1261,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 29 59 59 54 T T F <--- 1 020-039 32 32 30 59 59 56 T T F <--- 2 040-059 32 32 30 59 59 55 F F F <--- 3 060-079 32 32 30 59 59 56 F F F <--- 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 30 30 30 56 56 56 F F F 9 180-199 30 30 30 56 56 56 F F F 10 200-219 29 29 29 54 54 54 T T F <--- 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 30 59 59 55 F F F <--- 13 260-279 30 30 30 55 55 55 T T F <--- 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 32 32 24 59 59 43 F F F <--- 17 340-359 25 25 25 47 47 47 F F F 18 360-379 24 24 24 45 45 45 T T F <--- 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 30 30 30 56 56 56 F F F 22 440-459 30 30 30 56 56 56 T T F <--- 23 460-479 19 19 19 33 33 33 F F F 24 480-499 29 29 29 54 54 54 T T F <--- 25 500-519 30 30 30 56 56 56 T T F <--- 26 520-539 31 31 31 57 57 57 T T F <--- 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 32, Max flush: 59 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 33 33 33 60 60 60 F F F 1 020-039 33 33 33 60 60 60 F F F 2 040-059 32 32 32 59 59 59 F F F 3 060-079 32 32 32 59 59 59 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 30 30 30 56 56 56 F F F 9 180-199 30 30 30 56 56 56 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 33 33 33 60 60 60 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 32 32 32 59 59 59 F F F 17 340-359 25 25 25 47 47 47 F F F 18 360-379 33 33 33 60 60 60 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 30 30 30 56 56 56 F F F 22 440-459 33 33 33 60 60 60 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 33 33 33 60 60 60 F F F 26 520-539 33 33 33 60 60 60 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 33, Max flush: 60 Begin loop 17 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.115204059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.123444758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.123737069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.123762599Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.123771318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.123783278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Crucible c688aa40-598f-433c-b220-58e5b483ce03 has session id: 4dfb4ff8-d427-492c-8930-688b91293d70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.12407503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Upstairs opts: Upstairs UUID: c688aa40-598f-433c-b220-58e5b483ce03, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124104259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Crucible stats registered with UUID: c688aa40-598f-433c-b220-58e5b483ce03","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124115309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:07.124124618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124277334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124306283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124317313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124444839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"c688aa40-598f-433c-b220-58e5b483ce03 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124454199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124463349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124473429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124483338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124493088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124503208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124513128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"ds_connection connected from Ok(127.0.0.1:58708)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124574826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"ds_connection connected from Ok(127.0.0.1:65466)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124592675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"ds_connection connected from Ok(127.0.0.1:33744)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.124603565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125366694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125391693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125403622Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125675615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125700504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[0]R flush_numbers[0..12]: [60, 60, 59, 59, 54, 54, 54, 46, 56, 56, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125756763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[0]R generation[0..12]: [33, 33, 32, 32, 29, 29, 29, 24, 30, 30, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125767822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125779082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[1]R flush_numbers[0..12]: [60, 60, 59, 59, 54, 54, 54, 46, 56, 56, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125789652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[1]R generation[0..12]: [33, 33, 32, 32, 29, 29, 29, 24, 30, 30, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125800171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125811271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[2]R flush_numbers[0..12]: [60, 60, 59, 59, 54, 54, 54, 46, 56, 56, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125824691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[2]R generation[0..12]: [33, 33, 32, 32, 29, 29, 29, 24, 30, 30, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.12583515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.1258464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.12585627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125865929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Next flush: 61","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125875369Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125884879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"":"downstairs","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125894729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125903648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"0","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125913218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"1","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125923118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"client":"2","":"downstairs","session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125933048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125942297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"c688aa40-598f-433c-b220-58e5b483ce03 is now active with session: 4dfb4ff8-d427-492c-8930-688b91293d70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125952407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125961377Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"session_id":"4dfb4ff8-d427-492c-8930-688b91293d70"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125970557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Activated sub_volume c688aa40-598f-433c-b220-58e5b483ce03","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125980096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.125988216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.126325557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} 01/30 Write block 10 len 20480 data: 4 6 5 5 5 02/30 Write block 42 len 24576 data: 8 7 7 6 6 7 03/30 Write block 445 len 4096 data: 5 04/30 Read block 470 len 20480 05/30 Write block 57 len 36864 data: 4 3 3 3 3 3 4 4 6 06/30 Read block 352 len 8192 07/30 Read block 300 len 40960 08/30 Read block 30 len 40960 09/30 Flush 10/30 Write block 160 len 8192 data: 2 3 11/30 Read block 504 len 20480 12/30 Read block 494 len 28672 13/30 Write block 431 len 20480 data: 6 4 4 4 4 14/30 Read block 425 len 20480 15/30 Read block 109 len 16384 16/30 Write block 350 len 40960 data: 4 4 4 4 4 3 3 2 4 5 17/30 Write block 170 len 4096 data: 3 18/30 Read block 25 len 36864 19/30 Write block 185 len 28672 data: 4 4 5 7 7 6 6 20/30 Read block 57 len 20480 21/30 Read block 201 len 24576 22/30 Write block 321 len 36864 data: 5 4 5 5 4 5 5 5 7 23/30 Write block 332 len 16384 data: 5 4 4 5 24/30 Read block 209 len 16384 25/30 Read block 148 len 24576 26/30 Read block 445 len 32768 27/30 Read block 560 len 8192 28/30 Read block 119 len 28672 29/30 Write block 269 len 8192 data: 5 5 30/30 Write block 448 len 24576 data: 3 4 5 6 6 7 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Sent Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 1 Done Sent Done false 1003 Acked Read 5 Done Sent Done false 1004 Acked Write 9 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) None Some(JobId(1008)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:07.135243316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:07.135424601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1269,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 33 34 61 60 61 F F F <--- 1 020-039 33 33 33 60 60 60 F F F 2 040-059 34 32 34 61 59 61 F F F <--- 3 060-079 34 32 34 61 59 61 F F F <--- 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 30 30 30 56 56 56 T F T <--- 9 180-199 30 30 30 56 56 56 T F T <--- 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 33 33 33 60 60 60 T F T <--- 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 32 32 32 59 59 59 T F T <--- 17 340-359 25 25 25 47 47 47 T F T <--- 18 360-379 33 33 33 60 60 60 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 30 30 30 56 56 56 T F T <--- 22 440-459 34 33 34 61 60 61 T F T <--- 23 460-479 19 19 19 33 33 33 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 33 33 33 60 60 60 F F F 26 520-539 33 33 33 60 60 60 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 34, Max flush: 61 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F F F 1 020-039 33 33 33 60 60 60 F F F 2 040-059 34 34 34 61 61 61 F F F 3 060-079 34 34 34 61 61 61 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 35 35 35 62 62 62 F F F 9 180-199 35 35 35 62 62 62 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 35 35 35 62 62 62 F F F 14 280-299 27 27 27 51 51 51 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 35 35 35 62 62 62 F F F 17 340-359 35 35 35 62 62 62 F F F 18 360-379 33 33 33 60 60 60 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 35 35 35 62 62 62 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 19 19 19 33 33 33 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 33 33 33 60 60 60 F F F 26 520-539 33 33 33 60 60 60 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 F F F Max gen: 35, Max flush: 62 Begin loop 18 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.387727844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.395857846Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396158287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396184257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396193117Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396205016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Crucible c3538860-7951-4715-a611-ddab2fd55e68 has session id: 31da539d-b95a-45aa-bc4f-1a7162cf6768","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396512438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Upstairs opts: Upstairs UUID: c3538860-7951-4715-a611-ddab2fd55e68, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396558916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Crucible stats registered with UUID: c3538860-7951-4715-a611-ddab2fd55e68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396569546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:11.396578666Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396735481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39676521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39677843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396931986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396945885Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"c3538860-7951-4715-a611-ddab2fd55e68 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396979634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.396988974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397013993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397023863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397033493Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397043343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"ds_connection connected from Ok(127.0.0.1:44760)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397057562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397067282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"ds_connection connected from Ok(127.0.0.1:48440)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397082352Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"ds_connection connected from Ok(127.0.0.1:62754)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397161549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397799631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.397820281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39783433Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398110513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398142592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[0]R flush_numbers[0..12]: [61, 60, 61, 61, 54, 54, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39819245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[0]R generation[0..12]: [34, 33, 34, 34, 29, 29, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39820399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.39821498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[1]R flush_numbers[0..12]: [61, 60, 61, 61, 54, 54, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398225439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[1]R generation[0..12]: [34, 33, 34, 34, 29, 29, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398235979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398246969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[2]R flush_numbers[0..12]: [61, 60, 61, 61, 54, 54, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398257388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[2]R generation[0..12]: [34, 33, 34, 34, 29, 29, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398267868Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398278768Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398288038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398297687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Next flush: 63","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398307047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398319307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"":"downstairs","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398329186Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398338056Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"0","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398347776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"1","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398357476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"client":"2","":"downstairs","session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398367385Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398376505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"c3538860-7951-4715-a611-ddab2fd55e68 is now active with session: 31da539d-b95a-45aa-bc4f-1a7162cf6768","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398386355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398395225Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"session_id":"31da539d-b95a-45aa-bc4f-1a7162cf6768"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398404284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Activated sub_volume c3538860-7951-4715-a611-ddab2fd55e68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398412204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398420254Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.398780954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} 01/30 Write block 29 len 28672 data: 4 4 4 4 4 6 6 02/30 Read block 314 len 28672 03/30 Read block 329 len 20480 04/30 Write block 431 len 4096 data: 7 05/30 Write block 370 len 8192 data: 3 3 06/30 Write block 367 len 40960 data: 3 3 4 4 4 3 3 4 4 4 07/30 Read block 434 len 40960 08/30 Read block 36 len 24576 09/30 Flush 10/30 Write block 335 len 24576 data: 6 5 4 2 2 2 11/30 Read block 368 len 40960 12/30 Read block 329 len 36864 13/30 Read block 235 len 4096 14/30 Read block 288 len 32768 15/30 Read block 370 len 36864 16/30 Write block 460 len 16384 data: 4 2 3 3 17/30 Write block 70 len 8192 data: 6 4 18/30 Read block 446 len 40960 19/30 Read block 31 len 4096 20/30 Write block 582 len 8192 data: 4 4 21/30 Read block 209 len 24576 22/30 Read block 571 len 32768 23/30 Write block 280 len 16384 data: 3 2 2 2 24/30 Write block 111 len 36864 data: 3 3 3 3 4 5 5 5 5 25/30 Read block 404 len 16384 26/30 Read block 88 len 32768 27/30 Read block 486 len 24576 28/30 Write block 323 len 16384 data: 6 6 5 6 29/30 Read block 406 len 12288 30/30 Read block 350 len 8192 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 0 27 Done 29 4 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) None Some(JobId(1008)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:11.407664054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:11.407835899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1277,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F F F 1 020-039 36 33 36 63 60 63 F T F <--- 2 040-059 34 34 34 61 61 61 F F F 3 060-079 34 34 34 61 61 61 T F T <--- 4 080-099 29 29 29 54 54 54 F F F 5 100-119 29 29 29 54 54 54 T F T <--- 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 35 35 35 62 62 62 F F F 9 180-199 35 35 35 62 62 62 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 35 35 35 62 62 62 F F F 14 280-299 27 27 27 51 51 51 T F T <--- 15 300-319 17 17 17 29 29 29 F F F 16 320-339 35 35 35 62 62 62 T F T <--- 17 340-359 35 35 35 62 62 62 T F T <--- 18 360-379 36 33 36 63 60 63 F F F <--- 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 36 35 36 63 62 63 F T F <--- 22 440-459 35 35 35 62 62 62 F F F 23 460-479 19 19 19 33 33 33 T F T <--- 24 480-499 33 33 33 60 60 60 F F F 25 500-519 33 33 33 60 60 60 F F F 26 520-539 33 33 33 60 60 60 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 26 26 26 50 50 50 T F T <--- Max gen: 36, Max flush: 63 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F F F 1 020-039 36 36 36 63 63 63 F F F 2 040-059 34 34 34 61 61 61 F F F 3 060-079 37 37 37 64 64 64 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 37 37 37 64 64 64 F F F 6 120-139 29 29 29 54 54 54 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 35 35 35 62 62 62 F F F 9 180-199 35 35 35 62 62 62 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 35 35 35 62 62 62 F F F 14 280-299 37 37 37 64 64 64 F F F 15 300-319 17 17 17 29 29 29 F F F 16 320-339 37 37 37 64 64 64 F F F 17 340-359 37 37 37 64 64 64 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 30 30 30 55 55 55 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 37 37 37 64 64 64 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 33 33 33 60 60 60 F F F 26 520-539 33 33 33 60 60 60 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 F F F 29 580-599 37 37 37 64 64 64 F F F Max gen: 37, Max flush: 64 Begin loop 19 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.659205709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66735448Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.667661881Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.667686621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66769864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66771041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Crucible 110637da-ef7c-4cf2-973a-6645f7e4bdf1 has session id: 87c11eb7-3f34-4e7a-a20d-63ded941c2c6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668034861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Upstairs opts: Upstairs UUID: 110637da-ef7c-4cf2-973a-6645f7e4bdf1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668096279Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Crucible stats registered with UUID: 110637da-ef7c-4cf2-973a-6645f7e4bdf1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668107089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:15.668116309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668266364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668296674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668307633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66841692Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66843037Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"110637da-ef7c-4cf2-973a-6645f7e4bdf1 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668465529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668475209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668484948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668494568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668506708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668516887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668526897Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:64393)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668597035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:60135)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668733261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:60584)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.668759061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669323195Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669381013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669402133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669649706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669659575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[0]R flush_numbers[0..12]: [61, 63, 61, 64, 54, 64, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669696554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[0]R generation[0..12]: [34, 36, 34, 37, 29, 37, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669707584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669718744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[1]R flush_numbers[0..12]: [61, 63, 61, 64, 54, 64, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669729213Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[1]R generation[0..12]: [34, 36, 34, 37, 29, 37, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669741693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669752603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[2]R flush_numbers[0..12]: [61, 63, 61, 64, 54, 64, 54, 46, 62, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669763012Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[2]R generation[0..12]: [34, 36, 34, 37, 29, 37, 29, 24, 35, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669773432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669784322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669793701Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669803331Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Next flush: 65","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669812491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669821861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"":"downstairs","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66983169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66984061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"0","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66985014Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"1","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.66985982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"client":"2","":"downstairs","session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669869619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669878819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"110637da-ef7c-4cf2-973a-6645f7e4bdf1 is now active with session: 87c11eb7-3f34-4e7a-a20d-63ded941c2c6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669890259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669899169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"session_id":"87c11eb7-3f34-4e7a-a20d-63ded941c2c6"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669908288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Activated sub_volume 110637da-ef7c-4cf2-973a-6645f7e4bdf1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669916598Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.669924748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.670263058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} 01/30 Write block 100 len 12288 data: 3 3 3 02/30 Write block 509 len 4096 data: 3 03/30 Write block 163 len 12288 data: 4 4 3 04/30 Write block 51 len 36864 data: 5 4 4 3 3 3 5 4 4 05/30 Read block 458 len 36864 06/30 Read block 525 len 32768 07/30 Read block 484 len 8192 08/30 Read block 489 len 16384 09/30 Read block 85 len 12288 10/30 Read block 377 len 12288 11/30 Write block 125 len 24576 data: 5 5 6 5 3 3 12/30 Write block 264 len 28672 data: 4 5 4 4 4 6 6 13/30 Flush 14/30 Flush 15/30 Write block 310 len 24576 data: 2 2 2 2 2 2 16/30 Read block 228 len 20480 17/30 Write block 56 len 24576 data: 4 6 5 5 4 4 18/30 Write block 406 len 24576 data: 4 4 4 4 3 4 19/30 Write block 519 len 16384 data: 7 7 7 5 20/30 Write block 351 len 8192 data: 5 5 21/30 Write block 268 len 12288 data: 5 7 7 22/30 Read block 190 len 28672 23/30 Read block 288 len 20480 24/30 Flush 25/30 Write block 576 len 4096 data: 2 26/30 Read block 170 len 8192 27/30 Read block 556 len 40960 28/30 Write block 292 len 8192 data: 4 4 29/30 Read block 134 len 4096 30/30 Write block 352 len 12288 data: 6 5 5 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Sent false 1001 Acked Write 1 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:15.678116788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:15.678291183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1285,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F F F 1 020-039 36 36 36 63 63 63 F F F 2 040-059 38 38 34 67 67 61 F F F <--- 3 060-079 38 38 37 67 67 64 F F F <--- 4 080-099 29 29 29 54 54 54 F F F 5 100-119 38 38 37 65 65 64 F F F <--- 6 120-139 38 38 29 65 65 54 F F F <--- 7 140-159 24 24 24 46 46 46 F F F 8 160-179 38 38 35 65 65 62 F F F <--- 9 180-199 35 35 35 62 62 62 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 38 38 35 67 67 62 F F F <--- 14 280-299 37 37 37 64 64 64 T T F <--- 15 300-319 38 38 17 67 67 29 F F F <--- 16 320-339 37 37 37 64 64 64 F F F 17 340-359 38 38 37 67 67 64 T T F <--- 18 360-379 36 36 36 63 63 63 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 38 38 30 67 67 55 F F F <--- 21 420-439 36 36 36 63 63 63 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 37 37 37 64 64 64 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 38 38 33 67 67 60 F F F <--- 26 520-539 38 38 33 67 67 60 F F F <--- 27 540-559 31 31 31 57 57 57 F F F 28 560-579 31 31 31 57 57 57 T T F <--- 29 580-599 37 37 37 64 64 64 F F F Max gen: 38, Max flush: 67 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F F F 1 020-039 36 36 36 63 63 63 F F F 2 040-059 38 38 38 67 67 67 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 38 38 38 65 65 65 F F F 6 120-139 38 38 38 65 65 65 F F F 7 140-159 24 24 24 46 46 46 F F F 8 160-179 38 38 38 65 65 65 F F F 9 180-199 35 35 35 62 62 62 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 38 38 38 67 67 67 F F F 14 280-299 39 39 39 68 68 68 F F F 15 300-319 38 38 38 67 67 67 F F F 16 320-339 37 37 37 64 64 64 F F F 17 340-359 39 39 39 68 68 68 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 37 37 37 64 64 64 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 38 38 38 67 67 67 F F F 26 520-539 38 38 38 67 67 67 F F F 27 540-559 31 31 31 57 57 57 F F F 28 560-579 39 39 39 68 68 68 F F F 29 580-599 37 37 37 64 64 64 F F F Max gen: 39, Max flush: 68 Begin loop 20 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.929253823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.937758794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938061536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938090825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938099285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938111224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Crucible 39180874-6be9-40aa-b193-5133b3caaf23 has session id: 5f611c1b-f305-402c-867f-45368aa3f7da","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938430495Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Upstairs opts: Upstairs UUID: 39180874-6be9-40aa-b193-5133b3caaf23, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938469274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Crucible stats registered with UUID: 39180874-6be9-40aa-b193-5133b3caaf23","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938480144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:19.938491844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.93863733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938654279Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938664409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938792405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938806645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"39180874-6be9-40aa-b193-5133b3caaf23 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938830274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938839644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938849464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938859223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938868873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938878903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.938888832Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"ds_connection connected from Ok(127.0.0.1:50334)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.93899152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"ds_connection connected from Ok(127.0.0.1:42368)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.939016829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"ds_connection connected from Ok(127.0.0.1:42946)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.939028269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.939648161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.939668821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.93967978Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940011011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.94003403Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[0]R flush_numbers[0..12]: [61, 63, 67, 67, 54, 65, 65, 46, 65, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940077659Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[0]R generation[0..12]: [34, 36, 38, 38, 29, 38, 38, 24, 38, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940089209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940100368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[1]R flush_numbers[0..12]: [61, 63, 67, 67, 54, 65, 65, 46, 65, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940110818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[1]R generation[0..12]: [34, 36, 38, 38, 29, 38, 38, 24, 38, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940121178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940132038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[2]R flush_numbers[0..12]: [61, 63, 67, 67, 54, 65, 65, 46, 65, 62, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940142457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[2]R generation[0..12]: [34, 36, 38, 38, 29, 38, 38, 24, 38, 35, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940152887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940163797Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940176116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940185696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Next flush: 69","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940195076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940204575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"":"downstairs","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940214265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940223105Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"0","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940232745Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"1","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940242364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"client":"2","":"downstairs","session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940252274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940261484Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"39180874-6be9-40aa-b193-5133b3caaf23 is now active with session: 5f611c1b-f305-402c-867f-45368aa3f7da","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940271314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940279973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"session_id":"5f611c1b-f305-402c-867f-45368aa3f7da"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940288903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Activated sub_volume 39180874-6be9-40aa-b193-5133b3caaf23","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940296853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940304913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.940595964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} 01/30 Write block 503 len 12288 data: 3 3 3 02/30 Read block 272 len 16384 03/30 Write block 325 len 28672 data: 6 7 6 6 8 7 5 04/30 Read block 539 len 20480 05/30 Read block 204 len 8192 06/30 Read block 267 len 16384 07/30 Read block 120 len 36864 08/30 Write block 557 len 12288 data: 6 5 5 09/30 Read block 146 len 24576 10/30 Read block 295 len 32768 11/30 Read block 489 len 32768 12/30 Write block 38 len 16384 data: 6 7 7 7 13/30 Write block 185 len 12288 data: 5 5 6 14/30 Read block 129 len 20480 15/30 Write block 121 len 28672 data: 4 4 5 5 6 6 7 16/30 Write block 108 len 28672 data: 3 4 3 4 4 4 4 17/30 Write block 22 len 24576 data: 7 7 6 4 4 4 18/30 Read block 324 len 16384 19/30 Read block 524 len 24576 20/30 Read block 582 len 12288 21/30 Read block 509 len 12288 22/30 Write block 143 len 12288 data: 3 4 4 23/30 Write block 199 len 4096 data: 4 24/30 Write block 274 len 40960 data: 3 5 3 4 4 4 4 3 3 3 25/30 Read block 245 len 28672 26/30 Read block 78 len 32768 27/30 Write block 273 len 40960 data: 4 4 6 4 5 5 5 5 4 4 28/30 Read block 528 len 8192 29/30 Write block 13 len 28672 data: 6 6 5 5 3 4 6 30/30 Read block 87 len 20480 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:19.948957339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:19.949129835Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1293,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 61 61 61 F T T <--- 1 020-039 36 36 36 63 63 63 F T T <--- 2 040-059 38 38 38 67 67 67 F T T <--- 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 38 38 38 65 65 65 F T T <--- 6 120-139 38 38 38 65 65 65 F T T <--- 7 140-159 24 24 24 46 46 46 F T T <--- 8 160-179 38 38 38 65 65 65 F F F 9 180-199 35 35 35 62 62 62 F T T <--- 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 38 38 38 67 67 67 F T T <--- 14 280-299 39 39 39 68 68 68 F T T <--- 15 300-319 38 38 38 67 67 67 F F F 16 320-339 37 37 37 64 64 64 F T T <--- 17 340-359 39 39 39 68 68 68 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 37 37 37 64 64 64 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 38 38 38 67 67 67 F T T <--- 26 520-539 38 38 38 67 67 67 F F F 27 540-559 31 31 31 57 57 57 F T T <--- 28 560-579 39 39 39 68 68 68 F F F 29 580-599 37 37 37 64 64 64 F F F Max gen: 39, Max flush: 68 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 40 40 40 69 69 69 F F F 2 040-059 40 40 40 69 69 69 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 40 40 40 69 69 69 F F F 6 120-139 40 40 40 69 69 69 F F F 7 140-159 40 40 40 69 69 69 F F F 8 160-179 38 38 38 65 65 65 F F F 9 180-199 40 40 40 69 69 69 F F F 10 200-219 33 33 33 60 60 60 F F F 11 220-239 28 28 28 53 53 53 F F F 12 240-259 32 32 32 59 59 59 F F F 13 260-279 40 40 40 69 69 69 F F F 14 280-299 40 40 40 69 69 69 F F F 15 300-319 38 38 38 67 67 67 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 39 39 39 68 68 68 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 31 31 31 57 57 57 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 35 35 35 62 62 62 F F F 23 460-479 37 37 37 64 64 64 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 40 40 40 69 69 69 F F F 26 520-539 38 38 38 67 67 67 F F F 27 540-559 40 40 40 69 69 69 F F F 28 560-579 39 39 39 68 68 68 F F F 29 580-599 37 37 37 64 64 64 F F F Max gen: 40, Max flush: 69 Begin loop 21 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.206532865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.214818652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215113324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215138873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215147243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215158932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Crucible f362aae3-6df1-49d9-a085-947f1fe1ebc7 has session id: da8a0726-1db6-492d-b2a7-c0bed7054468","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215480723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Upstairs opts: Upstairs UUID: f362aae3-6df1-49d9-a085-947f1fe1ebc7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215510323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Crucible stats registered with UUID: f362aae3-6df1-49d9-a085-947f1fe1ebc7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215521272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:24.215530642Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215671348Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215702367Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215713417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215820734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215838203Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"f362aae3-6df1-49d9-a085-947f1fe1ebc7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215869812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215881642Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215891392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215901112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215910751Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215920871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.215930911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"ds_connection connected from Ok(127.0.0.1:37245)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216001619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"ds_connection connected from Ok(127.0.0.1:55057)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216019628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"ds_connection connected from Ok(127.0.0.1:63742)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216030798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216727848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216790107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.216810796Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217071099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217083298Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[0]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 69, 69, 69, 65, 69, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217103038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[0]R generation[0..12]: [40, 40, 40, 38, 29, 40, 40, 40, 38, 40, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217116087Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217127217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[1]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 69, 69, 69, 65, 69, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217137717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[1]R generation[0..12]: [40, 40, 40, 38, 29, 40, 40, 40, 38, 40, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217148177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217159126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[2]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 69, 69, 69, 65, 69, 60, 53]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217169466Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[2]R generation[0..12]: [40, 40, 40, 38, 29, 40, 40, 40, 38, 40, 33, 28]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217179886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217190955Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Max found gen is 41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217200385Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Generation requested: 42 >= found:41","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217210025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Next flush: 70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217219205Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217228504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"":"downstairs","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217238324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217246984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"0","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217256473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"1","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217267573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"client":"2","":"downstairs","session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217277333Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217286603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"f362aae3-6df1-49d9-a085-947f1fe1ebc7 is now active with session: da8a0726-1db6-492d-b2a7-c0bed7054468","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217296572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217305242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"session_id":"da8a0726-1db6-492d-b2a7-c0bed7054468"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217314282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Activated sub_volume f362aae3-6df1-49d9-a085-947f1fe1ebc7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217322522Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217330781Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.217644133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} 01/30 Write block 578 len 24576 data: 2 3 3 3 5 5 02/30 Read block 347 len 12288 03/30 Flush 04/30 Write block 239 len 40960 data: 4 4 5 4 4 4 3 4 3 4 05/30 Read block 245 len 16384 06/30 Write block 594 len 20480 data: 4 2 2 2 2 07/30 Read block 585 len 4096 08/30 Write block 347 len 28672 data: 4 4 5 5 6 7 6 09/30 Read block 385 len 24576 10/30 Read block 110 len 20480 11/30 Read block 95 len 40960 12/30 Write block 472 len 20480 data: 3 3 4 4 4 13/30 Read block 454 len 24576 14/30 Write block 216 len 24576 data: 4 4 4 3 5 6 15/30 Write block 119 len 4096 data: 6 16/30 Flush 17/30 Write block 306 len 40960 data: 4 3 3 3 3 3 3 3 3 3 18/30 Write block 341 len 4096 data: 2 19/30 Write block 440 len 40960 data: 3 3 2 3 5 6 5 3 4 5 20/30 Write block 396 len 4096 data: 4 21/30 Flush 22/30 Write block 152 len 40960 data: 4 4 3 4 4 3 3 2 3 4 23/30 Read block 389 len 24576 24/30 Read block 153 len 8192 25/30 Write block 534 len 20480 data: 4 4 5 6 6 26/30 Read block 398 len 36864 27/30 Read block 168 len 40960 28/30 Read block 416 len 32768 29/30 Read block 460 len 8192 30/30 Write block 119 len 16384 data: 7 4 5 5 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 10 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 1 26 Done 26 3 26 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1002)) Some(JobId(1020)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:24.227396938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:24.227548244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1301,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 40 40 40 69 69 69 F F F 2 040-059 40 40 40 69 69 69 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 42 40 42 71 69 71 T F T <--- 6 120-139 40 40 40 69 69 69 T F T <--- 7 140-159 40 40 40 69 69 69 T F T <--- 8 160-179 38 38 38 65 65 65 T F T <--- 9 180-199 40 40 40 69 69 69 F F F 10 200-219 42 33 42 71 60 71 F F F <--- 11 220-239 42 28 42 71 53 71 F T F <--- 12 240-259 42 32 42 71 59 71 F T F <--- 13 260-279 40 40 40 69 69 69 F F F 14 280-299 40 40 40 69 69 69 F F F 15 300-319 42 38 42 72 67 72 F F F <--- 16 320-339 40 40 40 69 69 69 F F F 17 340-359 42 39 42 72 68 72 F F F <--- 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 31 42 72 57 72 F F F <--- 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 42 35 42 72 62 72 F F F <--- 23 460-479 42 37 42 71 64 71 F F F <--- 24 480-499 33 33 33 60 60 60 F F F 25 500-519 40 40 40 69 69 69 F F F 26 520-539 38 38 38 67 67 67 T F T <--- 27 540-559 40 40 40 69 69 69 F F F 28 560-579 42 42 42 70 70 70 F F F 29 580-599 42 42 42 71 70 71 F T F <--- Max gen: 42, Max flush: 72 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 40 40 40 69 69 69 F F F 2 040-059 40 40 40 69 69 69 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 43 43 43 73 73 73 F F F 6 120-139 43 43 43 73 73 73 F F F 7 140-159 43 43 43 73 73 73 F F F 8 160-179 43 43 43 73 73 73 F F F 9 180-199 40 40 40 69 69 69 F F F 10 200-219 42 42 42 71 71 71 F F F 11 220-239 42 42 42 71 71 71 F F F 12 240-259 42 42 42 71 71 71 F F F 13 260-279 40 40 40 69 69 69 F F F 14 280-299 40 40 40 69 69 69 F F F 15 300-319 42 42 42 72 72 72 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 42 42 42 72 72 72 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 42 42 42 72 72 72 F F F 23 460-479 42 42 42 71 71 71 F F F 24 480-499 33 33 33 60 60 60 F F F 25 500-519 40 40 40 69 69 69 F F F 26 520-539 43 43 43 73 73 73 F F F 27 540-559 40 40 40 69 69 69 F F F 28 560-579 42 42 42 70 70 70 F F F 29 580-599 42 42 42 71 71 71 F F F Max gen: 43, Max flush: 73 Begin loop 22 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.484248824Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:28.491385704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.491436672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498291799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498561772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498597341Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498606881Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.49861878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Crucible 6ced8039-7ae5-411f-ab3c-51f78e983327 has session id: 92d5ecfb-7790-42cb-92b0-03bdf51c613c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498922612Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Upstairs opts: Upstairs UUID: 6ced8039-7ae5-411f-ab3c-51f78e983327, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498948771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Crucible stats registered with UUID: 6ced8039-7ae5-411f-ab3c-51f78e983327","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.498958541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:28.49896754Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499124346Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499141446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499151365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499287221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499301251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"6ced8039-7ae5-411f-ab3c-51f78e983327 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.4993271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.49933694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.49934679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.49935644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499365989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499375999Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499407508Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37329)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499477356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46087)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499489886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"ds_connection connected from Ok(127.0.0.1:35065)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.499567364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500212115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500254544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500273324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500538836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500550286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[0]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 73, 73, 73, 73, 69, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500574725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[0]R generation[0..12]: [40, 40, 40, 38, 29, 43, 43, 43, 43, 40, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500585605Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500596615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[1]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 73, 73, 73, 73, 69, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500607044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[1]R generation[0..12]: [40, 40, 40, 38, 29, 43, 43, 43, 43, 40, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500617354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500630894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[2]R flush_numbers[0..12]: [69, 69, 69, 67, 54, 73, 73, 73, 73, 69, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500641313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[2]R generation[0..12]: [40, 40, 40, 38, 29, 43, 43, 43, 43, 40, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500652013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500663013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500672503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500682262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Next flush: 74","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500691622Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500701132Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"":"downstairs","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500710981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500719631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"0","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500729251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"1","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500738911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"client":"2","":"downstairs","session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.50074866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.50075818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"6ced8039-7ae5-411f-ab3c-51f78e983327 is now active with session: 92d5ecfb-7790-42cb-92b0-03bdf51c613c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.50076822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.50077691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"session_id":"92d5ecfb-7790-42cb-92b0-03bdf51c613c"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500787229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Activated sub_volume 6ced8039-7ae5-411f-ab3c-51f78e983327","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500795379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.500803529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.50113464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} 01/30 Write block 233 len 32768 data: 4 5 5 3 3 3 5 5 02/30 Read block 323 len 16384 03/30 Write block 172 len 16384 data: 3 3 3 3 04/30 Read block 146 len 12288 05/30 Read block 14 len 12288 06/30 Write block 47 len 32768 data: 8 7 7 7 6 5 5 4 07/30 Read block 497 len 16384 08/30 Read block 95 len 12288 09/30 Read block 541 len 40960 10/30 Read block 528 len 28672 11/30 Read block 109 len 8192 12/30 Write block 441 len 36864 data: 4 3 4 6 7 6 4 5 6 13/30 Read block 205 len 12288 14/30 Write block 487 len 16384 data: 3 4 4 4 15/30 Write block 246 len 20480 data: 5 4 5 5 5 16/30 Read block 321 len 28672 17/30 Write block 21 len 12288 data: 6 8 8 18/30 Read block 287 len 28672 19/30 Write block 177 len 40960 data: 3 2 3 3 3 3 4 5 6 6 20/30 Read block 288 len 40960 21/30 Write block 212 len 12288 data: 6 4 3 22/30 Write block 347 len 12288 data: 5 5 6 23/30 Write block 159 len 16384 data: 3 4 5 4 24/30 Read block 155 len 8192 25/30 Read block 295 len 8192 26/30 Read block 88 len 24576 27/30 Read block 215 len 36864 28/30 Read block 425 len 32768 29/30 Write block 108 len 16384 data: 4 5 4 5 30/30 Write block 456 len 40960 data: 8 8 6 4 5 3 4 4 3 3 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 5 28 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:28.510145246Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:28.510299562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1309,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 40 40 40 69 69 69 F T T <--- 2 040-059 40 40 40 69 69 69 F T T <--- 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 43 43 43 73 73 73 F T T <--- 6 120-139 43 43 43 73 73 73 F F F 7 140-159 43 43 43 73 73 73 F T T <--- 8 160-179 43 43 43 73 73 73 T T T 9 180-199 40 40 40 69 69 69 F T T <--- 10 200-219 42 42 42 71 71 71 F T T <--- 11 220-239 42 42 42 71 71 71 T T T 12 240-259 42 42 42 71 71 71 T T T 13 260-279 40 40 40 69 69 69 F F F 14 280-299 40 40 40 69 69 69 F F F 15 300-319 42 42 42 72 72 72 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 42 42 42 72 72 72 F T T <--- 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 42 42 42 72 72 72 F T T <--- 23 460-479 42 42 42 71 71 71 F T T <--- 24 480-499 33 33 33 60 60 60 F T T <--- 25 500-519 40 40 40 69 69 69 F F F 26 520-539 43 43 43 73 73 73 F F F 27 540-559 40 40 40 69 69 69 F F F 28 560-579 42 42 42 70 70 70 F F F 29 580-599 42 42 42 71 71 71 F F F Max gen: 43, Max flush: 73 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now Files /tmp/test_repair/test_repair_verify.data and /tmp/test_repair/test_repair_verify.data.last differ 163d162 < 3, 165c164 < 3, --- > 5, 167a167 > 4, 179a180 > 3, 181,182d181 < 1, < 2, 185a185 > 3, 248a249 > 5, 250c251,253 < 3, --- > 5, > 5, > 5, 253,255d255 < 4, < 4, < 4, diff found after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 44 44 44 74 74 74 F F F 2 040-059 44 44 44 74 74 74 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 44 44 44 74 74 74 F F F 6 120-139 43 43 43 73 73 73 F F F 7 140-159 44 44 44 74 74 74 F F F 8 160-179 44 44 44 74 74 74 F F F 9 180-199 44 44 44 74 74 74 F F F 10 200-219 44 44 44 74 74 74 F F F 11 220-239 44 44 44 74 74 74 F F F 12 240-259 44 44 44 74 74 74 F F F 13 260-279 40 40 40 69 69 69 F F F 14 280-299 40 40 40 69 69 69 F F F 15 300-319 42 42 42 72 72 72 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 38 38 38 67 67 67 F F F 21 420-439 36 36 36 63 63 63 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 44 44 44 74 74 74 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 40 40 40 69 69 69 F F F 26 520-539 43 43 43 73 73 73 F F F 27 540-559 40 40 40 69 69 69 F F F 28 560-579 42 42 42 70 70 70 F F F 29 580-599 42 42 42 71 71 71 F F F Max gen: 44, Max flush: 74 Begin loop 23 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.777542676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:32.784678845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.784733204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.791534272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.791803065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.791831054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.791839714Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.791851423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Crucible 2c2f1e04-ed81-469a-a886-7ac6221fe1a5 has session id: f7936ac9-a951-45d9-b27f-75e60a4a30e9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792153285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Upstairs opts: Upstairs UUID: 2c2f1e04-ed81-469a-a886-7ac6221fe1a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792183674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Crucible stats registered with UUID: 2c2f1e04-ed81-469a-a886-7ac6221fe1a5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792197234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:32.792206673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792381079Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792413268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792424407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792533864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792547524Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"2c2f1e04-ed81-469a-a886-7ac6221fe1a5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792622022Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792631511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792641231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792651061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792660801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.7926709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.79268102Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:48058)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792739868Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:44837)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792758608Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:63800)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.792771878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793500847Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793522636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793553256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793865367Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793875887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[0]R flush_numbers[0..12]: [69, 74, 74, 67, 54, 74, 73, 74, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793920385Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[0]R generation[0..12]: [40, 44, 44, 38, 29, 44, 43, 44, 44, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793931655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793942695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[1]R flush_numbers[0..12]: [69, 74, 74, 67, 54, 74, 73, 74, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793953354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[1]R generation[0..12]: [40, 44, 44, 38, 29, 44, 43, 44, 44, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793963854Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793974924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[2]R flush_numbers[0..12]: [69, 74, 74, 67, 54, 74, 73, 74, 74, 74, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793985363Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[2]R generation[0..12]: [40, 44, 44, 38, 29, 44, 43, 44, 44, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.793995753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794009463Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Max found gen is 45","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794019002Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Generation requested: 46 >= found:45","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794028722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Next flush: 75","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794038152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794047572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"":"downstairs","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794057481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794066331Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"0","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794076071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"1","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794085801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"client":"2","":"downstairs","session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.79409566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.79410512Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"2c2f1e04-ed81-469a-a886-7ac6221fe1a5 is now active with session: f7936ac9-a951-45d9-b27f-75e60a4a30e9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.79411519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.79412403Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"session_id":"f7936ac9-a951-45d9-b27f-75e60a4a30e9"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794133509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Activated sub_volume 2c2f1e04-ed81-469a-a886-7ac6221fe1a5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794141949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794150069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.794481979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} 01/30 Write block 456 len 24576 data: 9 9 7 5 6 4 02/30 Write block 411 len 8192 data: 5 4 03/30 Write block 174 len 8192 data: 4 4 04/30 Read block 434 len 24576 05/30 Flush 06/30 Write block 304 len 20480 data: 3 3 5 4 4 07/30 Read block 427 len 12288 08/30 Read block 348 len 32768 09/30 Read block 82 len 12288 10/30 Write block 123 len 4096 data: 6 11/30 Write block 43 len 36864 data: 8 8 7 7 9 8 8 8 7 12/30 Read block 284 len 12288 13/30 Write block 30 len 36864 data: 5 5 5 5 7 7 5 5 7 14/30 Write block 221 len 40960 data: 7 5 5 5 5 5 5 4 5 5 15/30 Write block 123 len 4096 data: 7 16/30 Flush 17/30 Write block 573 len 24576 data: 2 2 2 3 2 3 18/30 Read block 492 len 4096 19/30 Read block 205 len 36864 20/30 Write block 193 len 28672 data: 4 4 4 2 3 4 5 21/30 Write block 276 len 40960 data: 5 6 6 6 6 5 5 4 2 2 22/30 Write block 420 len 32768 data: 3 3 3 3 3 4 4 4 23/30 Read block 49 len 12288 24/30 Read block 139 len 28672 25/30 Write block 42 len 8192 data: 9 9 26/30 Write block 530 len 12288 data: 3 3 3 27/30 Write block 47 len 4096 data: 10 28/30 Write block 558 len 28672 data: 6 6 6 6 6 4 5 29/30 Write block 503 len 24576 data: 4 4 4 3 3 3 30/30 Write block 274 len 40960 data: 5 7 6 7 7 7 7 6 6 5 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 2 Sent Done Done false 1002 Acked Write 2 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 7 Sent Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 2 34 Done 1 27 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:32.802858084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:32.803037719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1320,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 44 46 46 74 76 76 F F F <--- 2 040-059 44 46 46 74 76 76 F T T <--- 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 44 44 44 74 74 74 F F F 6 120-139 43 46 46 73 76 76 F F F <--- 7 140-159 44 44 44 74 74 74 F F F 8 160-179 44 46 46 74 75 75 F F F <--- 9 180-199 44 44 44 74 74 74 F T T <--- 10 200-219 44 44 44 74 74 74 F F F 11 220-239 44 46 46 74 76 76 F F F <--- 12 240-259 44 44 44 74 74 74 F F F 13 260-279 40 40 40 69 69 69 F T T <--- 14 280-299 40 40 40 69 69 69 F T T <--- 15 300-319 42 46 46 72 76 76 F F F <--- 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 38 46 46 67 75 75 F F F <--- 21 420-439 36 36 36 63 63 63 F T T <--- 22 440-459 44 46 46 74 75 75 T F F <--- 23 460-479 44 46 46 74 75 75 T F F <--- 24 480-499 44 44 44 74 74 74 F F F 25 500-519 40 40 40 69 69 69 F T T <--- 26 520-539 43 43 43 73 73 73 F T T <--- 27 540-559 40 40 40 69 69 69 F T T <--- 28 560-579 42 42 42 70 70 70 F T T <--- 29 580-599 42 42 42 71 71 71 F F F Max gen: 46, Max flush: 76 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 46 46 46 76 76 76 F F F 2 040-059 47 47 47 77 77 77 F F F 3 060-079 38 38 38 67 67 67 F F F 4 080-099 29 29 29 54 54 54 F F F 5 100-119 44 44 44 74 74 74 F F F 6 120-139 46 46 46 76 76 76 F F F 7 140-159 44 44 44 74 74 74 F F F 8 160-179 46 46 46 75 75 75 F F F 9 180-199 47 47 47 77 77 77 F F F 10 200-219 44 44 44 74 74 74 F F F 11 220-239 46 46 46 76 76 76 F F F 12 240-259 44 44 44 74 74 74 F F F 13 260-279 47 47 47 77 77 77 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 36 36 36 63 63 63 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 46 46 46 75 75 75 F F F 21 420-439 47 47 47 77 77 77 F F F 22 440-459 46 46 46 75 75 75 F F F 23 460-479 46 46 46 75 75 75 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 47 47 47 77 77 77 F F F 26 520-539 47 47 47 77 77 77 F F F 27 540-559 47 47 47 77 77 77 F F F 28 560-579 47 47 47 77 77 77 F F F 29 580-599 42 42 42 71 71 71 F F F Max gen: 47, Max flush: 77 Begin loop 24 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.074872703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:37.081972663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.082028512Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.088817601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089099253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089135342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089144342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089155961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Crucible 202f707a-7419-46a7-9e69-2f31f8dad6d6 has session id: 5c53aa69-dfeb-49dc-a67a-c7c8ad43e189","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089489122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Upstairs opts: Upstairs UUID: 202f707a-7419-46a7-9e69-2f31f8dad6d6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089518671Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Crucible stats registered with UUID: 202f707a-7419-46a7-9e69-2f31f8dad6d6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089530051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:37.089539371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089686667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089711616Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089721986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089874881Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089892541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"202f707a-7419-46a7-9e69-2f31f8dad6d6 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.08992135Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.08993114Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089941109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089950949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089960679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089970569Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.089980658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"ds_connection connected from Ok(127.0.0.1:34935)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090049126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"ds_connection connected from Ok(127.0.0.1:61006)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090061726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"ds_connection connected from Ok(127.0.0.1:59470)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090097085Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090754267Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090779866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.090794315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091085217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091103677Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[0]R flush_numbers[0..12]: [69, 76, 77, 67, 54, 74, 76, 74, 75, 77, 74, 76]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091121636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[0]R generation[0..12]: [40, 46, 47, 38, 29, 44, 46, 44, 46, 47, 44, 46]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091132616Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091144056Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[1]R flush_numbers[0..12]: [69, 76, 77, 67, 54, 74, 76, 74, 75, 77, 74, 76]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091154615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[1]R generation[0..12]: [40, 46, 47, 38, 29, 44, 46, 44, 46, 47, 44, 46]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091165205Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091176485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[2]R flush_numbers[0..12]: [69, 76, 77, 67, 54, 74, 76, 74, 75, 77, 74, 76]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091187014Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[2]R generation[0..12]: [40, 46, 47, 38, 29, 44, 46, 44, 46, 47, 44, 46]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091197634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091208674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091218173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091228393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Next flush: 78","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091239923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091249453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"":"downstairs","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091259202Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091268072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"0","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091277872Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"1","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091287782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"client":"2","":"downstairs","session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091297651Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091306821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"202f707a-7419-46a7-9e69-2f31f8dad6d6 is now active with session: 5c53aa69-dfeb-49dc-a67a-c7c8ad43e189","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.091316981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.09132603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"session_id":"5c53aa69-dfeb-49dc-a67a-c7c8ad43e189"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.09133589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Activated sub_volume 202f707a-7419-46a7-9e69-2f31f8dad6d6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.09134409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.09135234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.09168167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} 01/30 Write block 237 len 20480 data: 4 4 6 6 6 02/30 Read block 423 len 28672 03/30 Read block 478 len 20480 04/30 Read block 74 len 8192 05/30 Read block 383 len 32768 06/30 Write block 25 len 16384 data: 5 5 5 5 07/30 Read block 84 len 12288 08/30 Write block 577 len 12288 data: 3 4 4 09/30 Write block 24 len 16384 data: 7 6 6 6 10/30 Write block 148 len 28672 data: 2 4 4 4 5 5 4 11/30 Read block 77 len 12288 12/30 Write block 186 len 32768 data: 7 7 8 8 7 7 5 5 13/30 Read block 241 len 28672 14/30 Read block 23 len 24576 15/30 Write block 402 len 16384 data: 5 4 4 4 16/30 Read block 448 len 40960 17/30 Write block 72 len 36864 data: 3 4 5 4 4 4 4 5 4 18/30 Write block 162 len 28672 data: 4 5 5 4 3 3 4 19/30 Write block 247 len 4096 data: 4 20/30 Read block 254 len 4096 21/30 Read block 570 len 36864 22/30 Flush 23/30 Flush 24/30 Write block 365 len 20480 data: 4 4 4 4 5 25/30 Read block 476 len 16384 26/30 Read block 271 len 20480 27/30 Read block 567 len 20480 28/30 Read block 431 len 20480 29/30 Write block 550 len 16384 data: 3 3 4 4 30/30 Read block 91 len 12288 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 3 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 0 28 Done 3 29 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:37.10060777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:37.100781775Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1328,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 46 48 48 76 78 78 F F F <--- 2 040-059 47 47 47 77 77 77 F F F 3 060-079 38 48 48 67 78 78 F F F <--- 4 080-099 29 48 48 54 78 78 F F F <--- 5 100-119 44 44 44 74 74 74 F F F 6 120-139 46 46 46 76 76 76 F F F 7 140-159 44 48 48 74 78 78 F F F <--- 8 160-179 46 48 48 75 78 78 F F F <--- 9 180-199 47 48 48 77 78 78 F F F <--- 10 200-219 44 44 44 74 74 74 F F F 11 220-239 46 48 48 76 78 78 T F F <--- 12 240-259 44 48 48 74 78 78 T F F <--- 13 260-279 47 47 47 77 77 77 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 36 36 36 63 63 63 F T T <--- 19 380-399 42 42 42 72 72 72 F F F 20 400-419 46 48 48 75 78 78 F F F <--- 21 420-439 47 47 47 77 77 77 F F F 22 440-459 46 46 46 75 75 75 F F F 23 460-479 46 46 46 75 75 75 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 47 47 47 77 77 77 F F F 26 520-539 47 47 47 77 77 77 F F F 27 540-559 47 47 47 77 77 77 F T T <--- 28 560-579 47 48 48 77 78 78 F F F <--- 29 580-599 42 42 42 71 71 71 F F F Max gen: 48, Max flush: 78 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 48 48 48 78 78 78 F F F 2 040-059 47 47 47 77 77 77 F F F 3 060-079 48 48 48 78 78 78 F F F 4 080-099 48 48 48 78 78 78 F F F 5 100-119 44 44 44 74 74 74 F F F 6 120-139 46 46 46 76 76 76 F F F 7 140-159 48 48 48 78 78 78 F F F 8 160-179 48 48 48 78 78 78 F F F 9 180-199 48 48 48 78 78 78 F F F 10 200-219 44 44 44 74 74 74 F F F 11 220-239 48 48 48 78 78 78 F F F 12 240-259 48 48 48 78 78 78 F F F 13 260-279 47 47 47 77 77 77 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 49 49 49 79 79 79 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 48 48 48 78 78 78 F F F 21 420-439 47 47 47 77 77 77 F F F 22 440-459 46 46 46 75 75 75 F F F 23 460-479 46 46 46 75 75 75 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 47 47 47 77 77 77 F F F 26 520-539 47 47 47 77 77 77 F F F 27 540-559 49 49 49 79 79 79 F F F 28 560-579 48 48 48 78 78 78 F F F 29 580-599 42 42 42 71 71 71 F F F Max gen: 49, Max flush: 79 Begin loop 25 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.358127157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.366408534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.366696516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.366725575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.366736545Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.366748174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Crucible c9943662-4598-489e-a937-2c8b6c12ee1c has session id: 4a10476c-f297-4699-8541-edbf7037603a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367074735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Upstairs opts: Upstairs UUID: c9943662-4598-489e-a937-2c8b6c12ee1c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367106544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Crucible stats registered with UUID: c9943662-4598-489e-a937-2c8b6c12ee1c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367117804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:41.367127214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367281489Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367299559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367309699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367423525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367441465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"c9943662-4598-489e-a937-2c8b6c12ee1c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367464944Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367474144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367483844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367495183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367504693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367514623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367524472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"ds_connection connected from Ok(127.0.0.1:54230)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367559302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44813)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.36761024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"ds_connection connected from Ok(127.0.0.1:49054)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.367661699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368174104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368191464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.36832964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368503315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368522844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[0]R flush_numbers[0..12]: [69, 78, 77, 78, 78, 74, 76, 78, 78, 78, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368603792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[0]R generation[0..12]: [40, 48, 47, 48, 48, 44, 46, 48, 48, 48, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368615122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368626212Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[1]R flush_numbers[0..12]: [69, 78, 77, 78, 78, 74, 76, 78, 78, 78, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368639161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[1]R generation[0..12]: [40, 48, 47, 48, 48, 44, 46, 48, 48, 48, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368649771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368660641Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[2]R flush_numbers[0..12]: [69, 78, 77, 78, 78, 74, 76, 78, 78, 78, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.36867111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[2]R generation[0..12]: [40, 48, 47, 48, 48, 44, 46, 48, 48, 48, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.36868146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.36869246Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368701789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368711499Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Next flush: 80","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368720859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368730389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"":"downstairs","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368740328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368749018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"0","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368758858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"1","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368768518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"client":"2","":"downstairs","session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368778237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368789097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"c9943662-4598-489e-a937-2c8b6c12ee1c is now active with session: 4a10476c-f297-4699-8541-edbf7037603a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368798937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368807516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"session_id":"4a10476c-f297-4699-8541-edbf7037603a"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368816746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Activated sub_volume c9943662-4598-489e-a937-2c8b6c12ee1c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368824856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.368832886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.369190206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} 01/30 Write block 513 len 32768 data: 5 6 5 5 5 6 8 8 02/30 Read block 34 len 28672 03/30 Read block 147 len 40960 04/30 Read block 288 len 20480 05/30 Flush 06/30 Write block 516 len 40960 data: 6 6 7 9 9 8 6 6 6 6 07/30 Write block 577 len 36864 data: 4 5 5 4 4 6 6 5 5 08/30 Write block 94 len 28672 data: 2 3 3 3 3 3 4 09/30 Read block 392 len 16384 10/30 Write block 467 len 40960 data: 3 3 3 4 3 4 4 5 5 5 11/30 Write block 165 len 20480 data: 5 4 4 5 3 12/30 Read block 193 len 8192 13/30 Read block 537 len 20480 14/30 Write block 131 len 8192 data: 2 2 15/30 Read block 533 len 16384 16/30 Write block 417 len 16384 data: 3 2 2 4 17/30 Write block 270 len 16384 data: 8 5 5 5 18/30 Read block 503 len 8192 19/30 Write block 99 len 24576 data: 4 5 4 4 3 4 20/30 Write block 88 len 40960 data: 3 3 3 3 2 2 3 4 4 4 21/30 Flush 22/30 Write block 187 len 20480 data: 8 9 9 8 8 23/30 Read block 555 len 40960 24/30 Read block 351 len 4096 25/30 Read block 421 len 4096 26/30 Read block 413 len 32768 27/30 Read block 44 len 36864 28/30 Write block 365 len 4096 data: 5 29/30 Read block 45 len 16384 30/30 Read block 385 len 40960 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Read 7 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Write 7 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:41.378155204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:41.378318689Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1336,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 48 48 48 78 78 78 F F F 2 040-059 47 47 47 77 77 77 F F F 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 48 81 81 78 F F F <--- 5 100-119 50 50 44 81 81 74 F F F <--- 6 120-139 50 50 46 81 81 76 F F F <--- 7 140-159 48 48 48 78 78 78 F F F 8 160-179 50 50 48 81 81 78 F F F <--- 9 180-199 48 48 48 78 78 78 T T F <--- 10 200-219 44 44 44 74 74 74 F F F 11 220-239 48 48 48 78 78 78 F F F 12 240-259 48 48 48 78 78 78 F F F 13 260-279 50 50 47 81 81 77 F F F <--- 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 49 49 49 79 79 79 T T F <--- 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 48 81 81 78 F F F <--- 21 420-439 50 50 47 81 81 77 F F F <--- 22 440-459 46 46 46 75 75 75 F F F 23 460-479 50 50 46 81 81 75 F F F <--- 24 480-499 44 44 44 74 74 74 F F F 25 500-519 50 50 47 81 81 77 F F F <--- 26 520-539 50 50 47 81 81 77 F F F <--- 27 540-559 49 49 49 79 79 79 F F F 28 560-579 50 50 48 81 81 78 F F F <--- 29 580-599 50 50 42 81 81 71 F F F <--- Max gen: 50, Max flush: 81 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 48 48 48 78 78 78 F F F 2 040-059 47 47 47 77 77 77 F F F 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 50 50 50 81 81 81 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 48 48 48 78 78 78 F F F 8 160-179 50 50 50 81 81 81 F F F 9 180-199 51 51 51 82 82 82 F F F 10 200-219 44 44 44 74 74 74 F F F 11 220-239 48 48 48 78 78 78 F F F 12 240-259 48 48 48 78 78 78 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 40 40 40 69 69 69 F F F 17 340-359 44 44 44 74 74 74 F F F 18 360-379 51 51 51 82 82 82 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 50 50 50 81 81 81 F F F 22 440-459 46 46 46 75 75 75 F F F 23 460-479 50 50 50 81 81 81 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 50 50 50 81 81 81 F F F 26 520-539 50 50 50 81 81 81 F F F 27 540-559 49 49 49 79 79 79 F F F 28 560-579 50 50 50 81 81 81 F F F 29 580-599 50 50 50 81 81 81 F F F Max gen: 51, Max flush: 82 Begin loop 26 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.633331816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.641447228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.64173364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.641762249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.641771329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.641783219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Crucible f4e079ad-3644-4be1-81a2-e1e96106fbed has session id: da0d2bb9-6723-4ffb-a615-01aec98edf34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642148889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Upstairs opts: Upstairs UUID: f4e079ad-3644-4be1-81a2-e1e96106fbed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642177018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Crucible stats registered with UUID: f4e079ad-3644-4be1-81a2-e1e96106fbed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642190487Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:45.642199567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642335743Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642352013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642362063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642500289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642514398Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"f4e079ad-3644-4be1-81a2-e1e96106fbed active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642532648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642541388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642550817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642560317Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642569747Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642579587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642589426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"ds_connection connected from Ok(127.0.0.1:60850)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642646045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"ds_connection connected from Ok(127.0.0.1:42901)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642657014Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"ds_connection connected from Ok(127.0.0.1:48765)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.642669444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643310496Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643433602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643469371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643601578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643703205Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[0]R flush_numbers[0..12]: [69, 78, 77, 78, 81, 81, 81, 78, 81, 82, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643832061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[0]R generation[0..12]: [40, 48, 47, 48, 50, 50, 50, 48, 50, 51, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643843501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643854781Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[1]R flush_numbers[0..12]: [69, 78, 77, 78, 81, 81, 81, 78, 81, 82, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.64386528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[1]R generation[0..12]: [40, 48, 47, 48, 50, 50, 50, 48, 50, 51, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.64387594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.64388689Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[2]R flush_numbers[0..12]: [69, 78, 77, 78, 81, 81, 81, 78, 81, 82, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643897349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[2]R generation[0..12]: [40, 48, 47, 48, 50, 50, 50, 48, 50, 51, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643907899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643921559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643930949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643940528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Next flush: 83","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643949838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643959328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"":"downstairs","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643969257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643978097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"0","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643987927Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"1","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.643997757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"client":"2","":"downstairs","session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644007756Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644017036Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"f4e079ad-3644-4be1-81a2-e1e96106fbed is now active with session: da0d2bb9-6723-4ffb-a615-01aec98edf34","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644026896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644035516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"session_id":"da0d2bb9-6723-4ffb-a615-01aec98edf34"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644044615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Activated sub_volume f4e079ad-3644-4be1-81a2-e1e96106fbed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644052425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644060465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.644410215Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} 01/30 Write block 326 len 16384 data: 8 7 7 9 02/30 Write block 567 len 36864 data: 5 5 3 4 3 2 3 3 3 03/30 Read block 419 len 8192 04/30 Flush 05/30 Write block 526 len 28672 data: 6 6 4 3 4 4 4 06/30 Write block 146 len 36864 data: 3 2 3 5 5 5 6 6 5 07/30 Read block 101 len 32768 08/30 Read block 364 len 16384 09/30 Write block 347 len 8192 data: 6 6 10/30 Flush 11/30 Write block 181 len 32768 data: 4 4 5 6 7 8 9 10 12/30 Read block 396 len 4096 13/30 Write block 47 len 4096 data: 11 14/30 Read block 489 len 16384 15/30 Read block 497 len 32768 16/30 Read block 112 len 28672 17/30 Read block 9 len 20480 18/30 Read block 30 len 8192 19/30 Write block 575 len 16384 data: 4 4 5 6 20/30 Write block 151 len 12288 data: 6 7 7 21/30 Write block 434 len 36864 data: 5 5 4 3 3 4 4 5 4 22/30 Write block 248 len 36864 data: 5 5 5 5 5 5 5 4 4 23/30 Write block 34 len 4096 data: 8 24/30 Flush 25/30 Write block 167 len 12288 data: 5 6 4 26/30 Read block 106 len 40960 27/30 Read block 2 len 24576 28/30 Read block 229 len 4096 29/30 Read block 44 len 16384 30/30 Read block 43 len 40960 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Write 9 Done Done Sent false 1002 Acked Read 2 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 10 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:45.652630774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:45.652806719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1344,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 52 52 48 85 85 78 F F F <--- 2 040-059 52 52 47 85 85 77 F F F <--- 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 50 50 50 81 81 81 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 48 85 85 78 F F F <--- 8 160-179 50 50 50 81 81 81 T T F <--- 9 180-199 52 52 51 85 85 82 F F F <--- 10 200-219 44 44 44 74 74 74 F F F 11 220-239 48 48 48 78 78 78 F F F 12 240-259 52 52 48 85 85 78 F F F <--- 13 260-279 50 50 50 81 81 81 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 52 52 40 83 83 69 F F F <--- 17 340-359 52 52 44 84 84 74 F F F <--- 18 360-379 51 51 51 82 82 82 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 52 52 50 85 85 81 F F F <--- 22 440-459 52 52 46 85 85 75 F F F <--- 23 460-479 50 50 50 81 81 81 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 50 50 50 81 81 81 F F F 26 520-539 52 52 50 84 84 81 F F F <--- 27 540-559 49 49 49 79 79 79 F F F 28 560-579 52 52 50 85 85 81 F F F <--- 29 580-599 50 50 50 81 81 81 F F F Max gen: 52, Max flush: 85 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 52 52 52 85 85 85 F F F 2 040-059 52 52 52 85 85 85 F F F 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 50 50 50 81 81 81 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 52 52 52 85 85 85 F F F 10 200-219 44 44 44 74 74 74 F F F 11 220-239 48 48 48 78 78 78 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 47 47 47 77 77 77 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 52 52 52 83 83 83 F F F 17 340-359 52 52 52 84 84 84 F F F 18 360-379 51 51 51 82 82 82 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 52 52 52 85 85 85 F F F 22 440-459 52 52 52 85 85 85 F F F 23 460-479 50 50 50 81 81 81 F F F 24 480-499 44 44 44 74 74 74 F F F 25 500-519 50 50 50 81 81 81 F F F 26 520-539 52 52 52 84 84 84 F F F 27 540-559 49 49 49 79 79 79 F F F 28 560-579 52 52 52 85 85 85 F F F 29 580-599 50 50 50 81 81 81 F F F Max gen: 53, Max flush: 86 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.903998243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912152594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912458865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912486615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912495204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912506924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Crucible 4e7a5ff0-c7df-4967-9a93-1669d36cbe9e has session id: 3667c4eb-30af-410a-9035-4874e3ec0b7d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912807516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Upstairs opts: Upstairs UUID: 4e7a5ff0-c7df-4967-9a93-1669d36cbe9e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912858544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Crucible stats registered with UUID: 4e7a5ff0-c7df-4967-9a93-1669d36cbe9e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.912869224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:49.912878424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91301787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913034549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913044829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913188265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913199065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"4e7a5ff0-c7df-4967-9a93-1669d36cbe9e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913213034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913221864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913231644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913246303Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913255933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913265933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913311071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:57292)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913336281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45685)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91337133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:56468)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913384979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913971263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.913986552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91406618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914322703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914371302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[0]R flush_numbers[0..12]: [69, 85, 85, 78, 81, 81, 81, 85, 86, 85, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914413011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[0]R generation[0..12]: [40, 52, 52, 48, 50, 50, 50, 52, 53, 52, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91442421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91443529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[1]R flush_numbers[0..12]: [69, 85, 85, 78, 81, 81, 81, 85, 86, 85, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.91444586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[1]R generation[0..12]: [40, 52, 52, 48, 50, 50, 50, 52, 53, 52, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914456259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914467259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[2]R flush_numbers[0..12]: [69, 85, 85, 78, 81, 81, 81, 85, 86, 85, 74, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914477839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[2]R generation[0..12]: [40, 52, 52, 48, 50, 50, 50, 52, 53, 52, 44, 48]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914488218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914499108Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914508568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914518208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Next flush: 87","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914527717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914537187Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"":"downstairs","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914547267Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914556176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"0","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914567666Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"1","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914577636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"client":"2","":"downstairs","session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914587676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914596925Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"4e7a5ff0-c7df-4967-9a93-1669d36cbe9e is now active with session: 3667c4eb-30af-410a-9035-4874e3ec0b7d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914606875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914615685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"session_id":"3667c4eb-30af-410a-9035-4874e3ec0b7d"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914624675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Activated sub_volume 4e7a5ff0-c7df-4967-9a93-1669d36cbe9e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914632654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914640744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.914957455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} 01/30 Flush 02/30 Write block 182 len 12288 data: 5 6 7 03/30 Write block 234 len 20480 data: 6 6 4 5 5 04/30 Read block 51 len 24576 05/30 Write block 347 len 28672 data: 7 7 7 6 7 8 7 06/30 Read block 491 len 24576 07/30 Read block 358 len 12288 08/30 Write block 422 len 16384 data: 4 4 4 5 09/30 Write block 472 len 32768 data: 5 5 6 6 6 3 2 2 10/30 Read block 358 len 12288 11/30 Write block 471 len 28672 data: 4 6 6 7 7 7 4 12/30 Write block 527 len 28672 data: 7 5 4 5 5 5 4 13/30 Read block 80 len 40960 14/30 Write block 502 len 8192 data: 3 5 15/30 Read block 532 len 20480 16/30 Write block 205 len 8192 data: 4 2 17/30 Read block 296 len 28672 18/30 Read block 282 len 32768 19/30 Write block 484 len 12288 data: 3 3 2 20/30 Read block 324 len 40960 21/30 Write block 45 len 40960 data: 8 8 12 9 9 9 8 6 6 5 22/30 Read block 53 len 4096 23/30 Read block 574 len 4096 24/30 Flush 25/30 Write block 293 len 20480 data: 5 3 3 4 4 26/30 Write block 113 len 4096 data: 5 27/30 Read block 500 len 20480 28/30 Write block 489 len 40960 data: 5 5 5 5 5 4 4 3 3 3 29/30 Read block 195 len 24576 30/30 Read block 290 len 20480 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 4 Done Done Done false 1008 Acked Write 8 Done Done Done false 1009 Acked Read 3 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 21 22 Done 29 28 8 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:49.924503527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:49.924675982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1352,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 52 52 52 85 85 85 F F F 2 040-059 54 54 52 88 88 85 F F F <--- 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 50 50 50 81 81 81 T T F <--- 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 54 54 52 88 88 85 F F T <--- 10 200-219 54 54 44 88 88 74 F F F <--- 11 220-239 54 54 48 88 88 78 F F T <--- 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 47 47 47 77 77 77 T T F <--- 15 300-319 46 46 46 76 76 76 F F F 16 320-339 52 52 52 83 83 83 F F F 17 340-359 54 54 52 88 88 84 F F T <--- 18 360-379 51 51 51 82 82 82 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 54 54 52 88 88 85 F F T <--- 22 440-459 52 52 52 85 85 85 F F F 23 460-479 54 54 50 88 88 81 F F T <--- 24 480-499 54 54 44 88 88 74 T T F <--- 25 500-519 54 54 50 88 88 81 F F F <--- 26 520-539 54 54 52 88 88 84 F F F <--- 27 540-559 49 49 49 79 79 79 F F F 28 560-579 52 52 52 85 85 85 F F F 29 580-599 50 50 50 81 81 81 F F F Max gen: 54, Max flush: 88 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 69 69 69 F F F 1 020-039 52 52 52 85 85 85 F F F 2 040-059 54 54 54 88 88 88 F F F 3 060-079 48 48 48 78 78 78 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 55 55 55 89 89 89 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 54 54 54 88 88 88 F F F 10 200-219 54 54 54 88 88 88 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 55 55 55 89 89 89 F F F 15 300-319 46 46 46 76 76 76 F F F 16 320-339 52 52 52 83 83 83 F F F 17 340-359 54 54 54 88 88 88 F F F 18 360-379 51 51 51 82 82 82 F F F 19 380-399 42 42 42 72 72 72 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 54 54 54 88 88 88 F F F 22 440-459 52 52 52 85 85 85 F F F 23 460-479 54 54 54 88 88 88 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 54 54 54 88 88 88 F F F 26 520-539 54 54 54 88 88 88 F F F 27 540-559 49 49 49 79 79 79 F F F 28 560-579 52 52 52 85 85 85 F F F 29 580-599 50 50 50 81 81 81 F F F Max gen: 55, Max flush: 89 Begin loop 28 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.182741634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.191315773Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.191639204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.191667513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.191676133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.191687782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Crucible ff56f6c0-7888-46e9-a17a-ebaca99ef1e3 has session id: 1f6faa4c-b467-44ac-99b4-71cda9aec46d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192019683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Upstairs opts: Upstairs UUID: ff56f6c0-7888-46e9-a17a-ebaca99ef1e3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192052272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Crucible stats registered with UUID: ff56f6c0-7888-46e9-a17a-ebaca99ef1e3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192062522Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:54.192071501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192223517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192239567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192249516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192364873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192374833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"ff56f6c0-7888-46e9-a17a-ebaca99ef1e3 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192392112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192400992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192410712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192420512Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192430191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192440131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192450131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63883)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192510959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"ds_connection connected from Ok(127.0.0.1:49145)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192580217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"ds_connection connected from Ok(127.0.0.1:52836)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.192604166Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193217049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193229299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193241939Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193520601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.1935301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[0]R flush_numbers[0..12]: [69, 85, 88, 78, 81, 89, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193573609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[0]R generation[0..12]: [40, 52, 54, 48, 50, 55, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193584819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193595979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[1]R flush_numbers[0..12]: [69, 85, 88, 78, 81, 89, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193606488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[1]R generation[0..12]: [40, 52, 54, 48, 50, 55, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193616828Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193627798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[2]R flush_numbers[0..12]: [69, 85, 88, 78, 81, 89, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193639667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[2]R generation[0..12]: [40, 52, 54, 48, 50, 55, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193650047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193661047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193670316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193679786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193689116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193698596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"":"downstairs","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193708255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193716925Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"0","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193726735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"1","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193736415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"client":"2","":"downstairs","session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193746104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193755034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"ff56f6c0-7888-46e9-a17a-ebaca99ef1e3 is now active with session: 1f6faa4c-b467-44ac-99b4-71cda9aec46d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193764924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193773614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"session_id":"1f6faa4c-b467-44ac-99b4-71cda9aec46d"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193784313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Activated sub_volume ff56f6c0-7888-46e9-a17a-ebaca99ef1e3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193792163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.193800173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.194174972Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} 01/30 Write block 435 len 16384 data: 6 5 4 4 02/30 Write block 4 len 20480 data: 4 3 4 5 5 03/30 Read block 406 len 24576 04/30 Read block 273 len 12288 05/30 Write block 582 len 12288 data: 7 7 6 06/30 Read block 362 len 24576 07/30 Read block 141 len 4096 08/30 Read block 271 len 8192 09/30 Read block 303 len 28672 10/30 Read block 330 len 16384 11/30 Write block 76 len 16384 data: 5 5 5 6 12/30 Read block 354 len 8192 13/30 Read block 153 len 32768 14/30 Read block 329 len 40960 15/30 Read block 459 len 36864 16/30 Flush 17/30 Write block 100 len 4096 data: 6 18/30 Write block 17 len 32768 data: 4 5 7 7 7 9 9 8 19/30 Read block 562 len 36864 20/30 Read block 572 len 36864 21/30 Read block 195 len 28672 22/30 Read block 242 len 24576 23/30 Write block 561 len 40960 data: 7 7 5 6 5 5 6 6 4 5 24/30 Write block 298 len 12288 data: 5 3 4 25/30 Read block 368 len 36864 26/30 Write block 320 len 32768 data: 4 6 5 7 7 7 9 8 27/30 Write block 552 len 28672 data: 5 5 5 6 6 7 7 28/30 Read block 347 len 4096 29/30 Write block 375 len 40960 data: 5 5 6 4 4 4 5 5 6 7 30/30 Read block 418 len 20480 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 10 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:54.203461081Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:54.203632916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1360,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 56 56 69 90 90 T T T <--- 1 020-039 52 52 52 85 85 85 F T T <--- 2 040-059 54 54 54 88 88 88 F F F 3 060-079 48 56 56 78 90 90 F F F <--- 4 080-099 50 50 50 81 81 81 F F F 5 100-119 55 55 55 89 89 89 F T T <--- 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 54 54 54 88 88 88 F F F 10 200-219 54 54 54 88 88 88 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 55 55 55 89 89 89 F T T <--- 15 300-319 46 46 46 76 76 76 F T T <--- 16 320-339 52 52 52 83 83 83 F T T <--- 17 340-359 54 54 54 88 88 88 F F F 18 360-379 51 51 51 82 82 82 F T T <--- 19 380-399 42 42 42 72 72 72 F T T <--- 20 400-419 50 50 50 81 81 81 F F F 21 420-439 54 56 56 88 90 90 T F F <--- 22 440-459 52 52 52 85 85 85 F F F 23 460-479 54 54 54 88 88 88 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 54 54 54 88 88 88 F F F 26 520-539 54 54 54 88 88 88 F F F 27 540-559 49 49 49 79 79 79 F T T <--- 28 560-579 52 52 52 85 85 85 F T T <--- 29 580-599 50 56 56 81 90 90 F F F <--- Max gen: 56, Max flush: 90 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 91 91 91 F F F 1 020-039 57 57 57 91 91 91 F F F 2 040-059 54 54 54 88 88 88 F F F 3 060-079 56 56 56 90 90 90 F F F 4 080-099 50 50 50 81 81 81 F F F 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 54 54 54 88 88 88 F F F 10 200-219 54 54 54 88 88 88 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 57 57 57 91 91 91 F F F 15 300-319 57 57 57 91 91 91 F F F 16 320-339 57 57 57 91 91 91 F F F 17 340-359 54 54 54 88 88 88 F F F 18 360-379 57 57 57 91 91 91 F F F 19 380-399 57 57 57 91 91 91 F F F 20 400-419 50 50 50 81 81 81 F F F 21 420-439 56 56 56 90 90 90 F F F 22 440-459 52 52 52 85 85 85 F F F 23 460-479 54 54 54 88 88 88 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 54 54 54 88 88 88 F F F 26 520-539 54 54 54 88 88 88 F F F 27 540-559 57 57 57 91 91 91 F F F 28 560-579 57 57 57 91 91 91 F F F 29 580-599 56 56 56 90 90 90 F F F Max gen: 57, Max flush: 91 Begin loop 29 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.465347496Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.473987543Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474285254Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474309354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474318113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474329923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Crucible 41712beb-cf85-4827-a215-8f7be14d9648 has session id: 51f5d736-0f7b-4b99-b8fd-3cdbe128a32e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474642904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Upstairs opts: Upstairs UUID: 41712beb-cf85-4827-a215-8f7be14d9648, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474672443Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Crucible stats registered with UUID: 41712beb-cf85-4827-a215-8f7be14d9648","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474683613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:58.474692893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474845319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474875408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474888807Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.474993524Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475007284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"41712beb-cf85-4827-a215-8f7be14d9648 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475041963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475051683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475061572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475071492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475081312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475091402Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475101401Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55641)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.47514923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54875)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.47516058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"ds_connection connected from Ok(127.0.0.1:35930)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475229518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475827771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.47585406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.475968947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476167791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476194271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[0]R flush_numbers[0..12]: [91, 91, 88, 90, 81, 91, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476244059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[0]R generation[0..12]: [57, 57, 54, 56, 50, 57, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476255389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476266589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[1]R flush_numbers[0..12]: [91, 91, 88, 90, 81, 91, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476277168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[1]R generation[0..12]: [57, 57, 54, 56, 50, 57, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476287798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476298888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[2]R flush_numbers[0..12]: [91, 91, 88, 90, 81, 91, 81, 85, 86, 88, 88, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476309437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[2]R generation[0..12]: [57, 57, 54, 56, 50, 57, 50, 52, 53, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476319857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476330827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476340297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476350206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Next flush: 92","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476359526Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476371956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"":"downstairs","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476381905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476390625Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"0","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476400345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"1","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476409975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"client":"2","":"downstairs","session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476419914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476429064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"41712beb-cf85-4827-a215-8f7be14d9648 is now active with session: 51f5d736-0f7b-4b99-b8fd-3cdbe128a32e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476439054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476447784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"session_id":"51f5d736-0f7b-4b99-b8fd-3cdbe128a32e"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476456843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Activated sub_volume 41712beb-cf85-4827-a215-8f7be14d9648","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476465013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476473313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.476809453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} 01/30 Write block 417 len 40960 data: 4 3 3 5 4 5 5 5 6 5 02/30 Read block 516 len 40960 03/30 Read block 447 len 40960 04/30 Read block 350 len 40960 05/30 Write block 81 len 40960 data: 3 3 4 4 3 3 3 4 4 4 06/30 Write block 87 len 16384 data: 4 5 5 5 07/30 Flush 08/30 Write block 575 len 32768 data: 5 5 6 7 6 5 5 8 09/30 Read block 89 len 16384 10/30 Write block 455 len 24576 data: 7 10 10 8 6 7 11/30 Flush 12/30 Write block 504 len 4096 data: 5 13/30 Write block 531 len 8192 data: 6 6 14/30 Read block 528 len 8192 15/30 Write block 371 len 36864 data: 5 4 4 5 6 6 7 5 5 16/30 Read block 327 len 40960 17/30 Read block 161 len 16384 18/30 Read block 58 len 28672 19/30 Write block 40 len 4096 data: 8 20/30 Write block 87 len 24576 data: 5 6 6 6 4 3 21/30 Read block 130 len 8192 22/30 Read block 396 len 8192 23/30 Read block 383 len 40960 24/30 Write block 429 len 28672 data: 5 6 8 5 5 6 7 25/30 Write block 355 len 28672 data: 4 4 3 5 6 4 4 26/30 Read block 433 len 36864 27/30 Write block 350 len 4096 data: 7 28/30 Read block 197 len 24576 29/30 Write block 199 len 32768 data: 6 4 4 4 5 4 5 3 30/30 Read block 575 len 8192 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 1 Done Done Done false 1012 Acked Write 2 Done Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 8 Sent Done Sent false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 0 2 19 Done 2 19 17 38 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:10:58.486074823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:10:58.486224429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1368,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 91 91 91 F F F 1 020-039 57 57 57 91 91 91 F F F 2 040-059 54 54 54 88 88 88 F T T <--- 3 060-079 56 56 56 90 90 90 F F F 4 080-099 58 58 58 92 92 92 F T T <--- 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 54 54 54 88 88 88 F T T <--- 10 200-219 54 54 54 88 88 88 F T T <--- 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 57 57 57 91 91 91 F F F 15 300-319 57 57 57 91 91 91 F F F 16 320-339 57 57 57 91 91 91 F F F 17 340-359 54 54 54 88 88 88 F T T <--- 18 360-379 57 57 57 91 91 91 F T T <--- 19 380-399 57 57 57 91 91 91 F F F 20 400-419 58 58 58 92 92 92 F F F 21 420-439 58 58 58 92 92 92 F T T <--- 22 440-459 58 58 58 93 93 93 F F F 23 460-479 58 58 58 93 93 93 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 54 54 54 88 88 88 T T T 26 520-539 54 54 54 88 88 88 T T T 27 540-559 57 57 57 91 91 91 F F F 28 560-579 58 58 58 93 93 93 F F F 29 580-599 58 58 58 93 93 93 F F F Max gen: 58, Max flush: 93 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 91 91 91 F F F 1 020-039 57 57 57 91 91 91 F F F 2 040-059 59 59 59 94 94 94 F F F 3 060-079 56 56 56 90 90 90 F F F 4 080-099 59 59 59 94 94 94 F F F 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 F F F 8 160-179 53 53 53 86 86 86 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 F F F 13 260-279 50 50 50 81 81 81 F F F 14 280-299 57 57 57 91 91 91 F F F 15 300-319 57 57 57 91 91 91 F F F 16 320-339 57 57 57 91 91 91 F F F 17 340-359 59 59 59 94 94 94 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 57 57 57 91 91 91 F F F 20 400-419 58 58 58 92 92 92 F F F 21 420-439 59 59 59 94 94 94 F F F 22 440-459 58 58 58 93 93 93 F F F 23 460-479 58 58 58 93 93 93 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 59 59 59 94 94 94 F F F 27 540-559 57 57 57 91 91 91 F F F 28 560-579 58 58 58 93 93 93 F F F 29 580-599 58 58 58 93 93 93 F F F Max gen: 59, Max flush: 94 Begin loop 30 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.740063759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74822653Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.748521301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.748546771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74855534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74856706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Crucible 36c95067-0f60-47bc-a4f8-e7b2bd54af54 has session id: 3c0b16e6-986e-4252-98cd-77dbd5ea8129","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Upstairs opts: Upstairs UUID: 36c95067-0f60-47bc-a4f8-e7b2bd54af54, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.748949779Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Crucible stats registered with UUID: 36c95067-0f60-47bc-a4f8-e7b2bd54af54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.748961319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:02.748970719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749127474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749157084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749167913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74928759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.74930152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"36c95067-0f60-47bc-a4f8-e7b2bd54af54 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749338878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749348378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749358108Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749367928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749377557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749387817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749399757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"ds_connection connected from Ok(127.0.0.1:32869)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749472835Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"ds_connection connected from Ok(127.0.0.1:55171)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749498284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"ds_connection connected from Ok(127.0.0.1:41858)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.749509604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750163885Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750209914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750227733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750467757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750488206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[0]R flush_numbers[0..12]: [91, 91, 94, 90, 94, 91, 81, 85, 86, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750506646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[0]R generation[0..12]: [57, 57, 59, 56, 59, 57, 50, 52, 53, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750517445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750528705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[1]R flush_numbers[0..12]: [91, 91, 94, 90, 94, 91, 81, 85, 86, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750539315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[1]R generation[0..12]: [57, 57, 59, 56, 59, 57, 50, 52, 53, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750550094Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750563704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[2]R flush_numbers[0..12]: [91, 91, 94, 90, 94, 91, 81, 85, 86, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750574444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[2]R generation[0..12]: [57, 57, 59, 56, 59, 57, 50, 52, 53, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750585083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750596273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750605973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750615713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Next flush: 95","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750625022Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750634492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"":"downstairs","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750644702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750653791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"0","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750663861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"1","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750673921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"client":"2","":"downstairs","session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750684071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.7506934Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"36c95067-0f60-47bc-a4f8-e7b2bd54af54 is now active with session: 3c0b16e6-986e-4252-98cd-77dbd5ea8129","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.75070395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.7507144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"session_id":"3c0b16e6-986e-4252-98cd-77dbd5ea8129"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.75072358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Activated sub_volume 36c95067-0f60-47bc-a4f8-e7b2bd54af54","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750731809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.750740109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.751042151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} 01/30 Write block 556 len 4096 data: 7 02/30 Read block 473 len 8192 03/30 Read block 589 len 12288 04/30 Read block 539 len 24576 05/30 Flush 06/30 Write block 299 len 12288 data: 4 5 4 07/30 Read block 126 len 32768 08/30 Write block 337 len 8192 data: 5 3 09/30 Flush 10/30 Write block 392 len 32768 data: 5 5 4 4 5 4 4 4 11/30 Flush 12/30 Write block 339 len 40960 data: 3 3 3 2 2 3 3 4 8 8 13/30 Flush 14/30 Write block 549 len 32768 data: 3 4 4 6 6 6 7 8 15/30 Write block 162 len 32768 data: 5 6 6 6 5 6 7 5 16/30 Read block 459 len 4096 17/30 Write block 408 len 32768 data: 5 5 4 6 5 4 4 4 18/30 Read block 110 len 28672 19/30 Write block 317 len 32768 data: 4 4 5 5 7 6 8 8 20/30 Write block 40 len 20480 data: 9 8 10 10 9 21/30 Write block 25 len 8192 data: 7 7 22/30 Write block 339 len 20480 data: 4 4 4 3 3 23/30 Read block 293 len 16384 24/30 Read block 453 len 20480 25/30 Read block 583 len 8192 26/30 Write block 148 len 40960 data: 4 6 6 7 8 8 6 5 5 4 27/30 Write block 270 len 28672 data: 9 6 6 6 6 8 7 28/30 Write block 81 len 12288 data: 4 4 5 29/30 Write block 259 len 36864 data: 4 4 4 4 3 5 6 5 5 30/30 Write block 160 len 28672 data: 4 5 6 7 7 7 6 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:02.759483723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:02.759644089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1376,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 91 91 91 F F F 1 020-039 57 57 57 91 91 91 T F T <--- 2 040-059 59 59 59 94 94 94 T F T <--- 3 060-079 56 56 56 90 90 90 F F F 4 080-099 59 59 59 94 94 94 T F T <--- 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 52 52 52 85 85 85 T F T <--- 8 160-179 53 53 53 86 86 86 T F T <--- 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 52 52 52 85 85 85 T F T <--- 13 260-279 50 50 50 81 81 81 T F T <--- 14 280-299 60 57 60 96 91 96 F F F <--- 15 300-319 60 57 60 96 91 96 T F T <--- 16 320-339 60 57 60 98 91 98 T F T <--- 17 340-359 60 59 60 98 94 98 T F T <--- 18 360-379 59 59 59 94 94 94 F F F 19 380-399 60 57 60 97 91 97 F F F <--- 20 400-419 58 58 58 92 92 92 T F T <--- 21 420-439 59 59 59 94 94 94 F F F 22 440-459 58 58 58 93 93 93 F F F 23 460-479 58 58 58 93 93 93 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 59 59 59 94 94 94 F F F 27 540-559 60 57 60 95 91 95 T T T <--- 28 560-579 58 58 58 93 93 93 F F F 29 580-599 58 58 58 93 93 93 F F F Max gen: 60, Max flush: 98 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 91 91 91 F F F 1 020-039 61 61 61 99 99 99 F F F 2 040-059 61 61 61 99 99 99 F F F 3 060-079 56 56 56 90 90 90 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 61 61 61 99 99 99 F F F 8 160-179 61 61 61 99 99 99 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 61 61 61 99 99 99 F F F 13 260-279 61 61 61 99 99 99 F F F 14 280-299 60 60 60 96 96 96 F F F 15 300-319 61 61 61 99 99 99 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 60 60 60 97 97 97 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 59 59 59 94 94 94 F F F 22 440-459 58 58 58 93 93 93 F F F 23 460-479 58 58 58 93 93 93 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 59 59 59 94 94 94 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 58 58 58 93 93 93 F F F 29 580-599 58 58 58 93 93 93 F F F Max gen: 61, Max flush: 99 Begin loop 31 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.019921628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.0280279Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028318472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028342641Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028351361Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.02836321Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Crucible 698e0dc0-e9c8-45e1-bc0f-85613d09a829 has session id: 1949a081-89a9-4df0-baa4-199bef842995","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.02871437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Upstairs opts: Upstairs UUID: 698e0dc0-e9c8-45e1-bc0f-85613d09a829, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028770709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Crucible stats registered with UUID: 698e0dc0-e9c8-45e1-bc0f-85613d09a829","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028781319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:07.028790558Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028961434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.028991523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029002672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.02909543Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029109689Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"698e0dc0-e9c8-45e1-bc0f-85613d09a829 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029145128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029154618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029164428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029174278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029183887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029194107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029204237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"ds_connection connected from Ok(127.0.0.1:57486)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029237376Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"ds_connection connected from Ok(127.0.0.1:43906)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029263625Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"ds_connection connected from Ok(127.0.0.1:65239)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029275045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029929236Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029951286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.029962555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030252207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030262237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[0]R flush_numbers[0..12]: [91, 99, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030301536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[0]R generation[0..12]: [57, 61, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030312826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030324215Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[1]R flush_numbers[0..12]: [91, 99, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030334725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[1]R generation[0..12]: [57, 61, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030345275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030356444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[2]R flush_numbers[0..12]: [91, 99, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030367044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[2]R generation[0..12]: [57, 61, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030377544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030388793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030398383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030410623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Next flush: 100","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030420143Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030429622Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"":"downstairs","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030439532Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030448442Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"0","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030458121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"1","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030467981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"client":"2","":"downstairs","session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030478001Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030487211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"698e0dc0-e9c8-45e1-bc0f-85613d09a829 is now active with session: 1949a081-89a9-4df0-baa4-199bef842995","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.03049735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.03050662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"session_id":"1949a081-89a9-4df0-baa4-199bef842995"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.03051587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Activated sub_volume 698e0dc0-e9c8-45e1-bc0f-85613d09a829","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.03052406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.030532289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.03087577Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} 01/30 Write block 423 len 16384 data: 6 6 7 6 02/30 Flush 03/30 Write block 462 len 24576 data: 5 5 4 4 3 4 04/30 Read block 463 len 8192 05/30 Read block 222 len 4096 06/30 Write block 7 len 4096 data: 6 07/30 Flush 08/30 Flush 09/30 Write block 577 len 24576 data: 7 8 7 6 6 9 10/30 Read block 67 len 4096 11/30 Read block 419 len 20480 12/30 Read block 3 len 28672 13/30 Read block 82 len 16384 14/30 Read block 503 len 16384 15/30 Flush 16/30 Write block 12 len 40960 data: 6 7 7 6 6 5 6 8 8 8 17/30 Read block 159 len 12288 18/30 Read block 117 len 40960 19/30 Write block 476 len 8192 data: 8 5 20/30 Read block 65 len 40960 21/30 Write block 437 len 32768 data: 5 5 5 5 6 5 5 7 22/30 Read block 64 len 16384 23/30 Read block 560 len 32768 24/30 Read block 241 len 20480 25/30 Write block 536 len 8192 data: 6 7 26/30 Read block 246 len 4096 27/30 Read block 103 len 4096 28/30 Write block 277 len 4096 data: 8 29/30 Write block 445 len 4096 data: 8 30/30 Read block 76 len 28672 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 6 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Read 1 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 28 29 Done 27 28 0 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:07.039481698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:07.039669563Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1384,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 57 101 101 91 T T F <--- 1 020-039 61 61 61 99 99 99 T T F <--- 2 040-059 61 61 61 99 99 99 F F F 3 060-079 56 56 56 90 90 90 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 61 61 61 99 99 99 F F F 8 160-179 61 61 61 99 99 99 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 61 61 61 99 99 99 F F F 13 260-279 61 61 61 99 99 99 T T F <--- 14 280-299 60 60 60 96 96 96 F F F 15 300-319 61 61 61 99 99 99 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 60 60 60 97 97 97 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 62 62 62 100 100 100 T T F <--- 22 440-459 58 58 58 93 93 93 T T F <--- 23 460-479 62 62 58 101 101 93 T T F <--- 24 480-499 55 55 55 89 89 89 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 59 59 59 94 94 94 T T F <--- 27 540-559 61 61 61 99 99 99 F F F 28 560-579 62 62 58 103 103 93 F F F <--- 29 580-599 62 62 58 103 103 93 F F F <--- Max gen: 62, Max flush: 103 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 63 63 63 104 104 104 F F F 1 020-039 63 63 63 104 104 104 F F F 2 040-059 61 61 61 99 99 99 F F F 3 060-079 56 56 56 90 90 90 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 57 57 57 91 91 91 F F F 6 120-139 50 50 50 81 81 81 F F F 7 140-159 61 61 61 99 99 99 F F F 8 160-179 61 61 61 99 99 99 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 61 61 61 99 99 99 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 60 60 60 96 96 96 F F F 15 300-319 61 61 61 99 99 99 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 60 60 60 97 97 97 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 63 63 63 104 104 104 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 55 55 55 89 89 89 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 62 62 62 103 103 103 F F F 29 580-599 62 62 62 103 103 103 F F F Max gen: 63, Max flush: 104 Begin loop 32 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.286202358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.294362479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.294645711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.29467476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.29468621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.29469799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Crucible 04b47145-7b10-4bb3-96d0-4b7c9347aab4 has session id: 37c74ec0-5222-4d89-98da-536c4f66e24e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295098158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Upstairs opts: Upstairs UUID: 04b47145-7b10-4bb3-96d0-4b7c9347aab4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295127917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Crucible stats registered with UUID: 04b47145-7b10-4bb3-96d0-4b7c9347aab4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295138937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:11.295148177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295289493Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295308082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295318342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295443579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295461178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"04b47145-7b10-4bb3-96d0-4b7c9347aab4 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295488427Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295498147Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295507947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295519266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295528806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295538886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295549036Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"ds_connection connected from Ok(127.0.0.1:34665)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295596594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"ds_connection connected from Ok(127.0.0.1:46450)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295608454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64203)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.295618774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296312224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296338313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296405742Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296644855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296690644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[0]R flush_numbers[0..12]: [104, 104, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296737822Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[0]R generation[0..12]: [63, 63, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296748702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296762202Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[1]R flush_numbers[0..12]: [104, 104, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296772621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[1]R generation[0..12]: [63, 63, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296783081Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296794011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[2]R flush_numbers[0..12]: [104, 104, 99, 90, 99, 91, 81, 99, 99, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.29680443Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[2]R generation[0..12]: [63, 63, 61, 56, 61, 57, 50, 61, 61, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.2968149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.29682603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296835519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296845209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Next flush: 105","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296854729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296864119Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"":"downstairs","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296874068Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296882988Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"0","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296892848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"1","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296902578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"client":"2","":"downstairs","session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296914407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296923447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"04b47145-7b10-4bb3-96d0-4b7c9347aab4 is now active with session: 37c74ec0-5222-4d89-98da-536c4f66e24e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296933427Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296941946Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"session_id":"37c74ec0-5222-4d89-98da-536c4f66e24e"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296951026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Activated sub_volume 04b47145-7b10-4bb3-96d0-4b7c9347aab4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296958886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.296966966Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.297329636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} 01/30 Write block 141 len 24576 data: 5 4 4 5 5 4 02/30 Write block 176 len 4096 data: 3 03/30 Write block 101 len 8192 data: 5 5 04/30 Flush 05/30 Write block 286 len 36864 data: 3 3 2 2 3 3 5 6 4 06/30 Write block 31 len 28672 data: 6 6 6 9 8 6 6 07/30 Read block 225 len 4096 08/30 Read block 300 len 36864 09/30 Write block 53 len 32768 data: 7 6 4 5 7 6 6 5 10/30 Write block 442 len 24576 data: 6 6 8 9 7 5 11/30 Read block 221 len 8192 12/30 Read block 51 len 8192 13/30 Write block 393 len 12288 data: 6 5 5 14/30 Read block 74 len 32768 15/30 Read block 259 len 32768 16/30 Read block 564 len 4096 17/30 Read block 296 len 4096 18/30 Write block 118 len 20480 data: 6 8 5 6 6 19/30 Write block 155 len 8192 data: 6 6 20/30 Read block 139 len 40960 21/30 Write block 55 len 20480 data: 5 6 8 7 7 22/30 Write block 148 len 8192 data: 5 7 23/30 Read block 278 len 36864 24/30 Write block 293 len 4096 data: 7 25/30 Write block 0 len 36864 data: 2 2 3 3 5 4 5 7 6 26/30 Read block 50 len 4096 27/30 Write block 117 len 40960 data: 6 7 9 6 7 7 8 6 7 7 28/30 Read block 576 len 32768 29/30 Read block 592 len 28672 30/30 Write block 496 len 12288 data: 4 4 4 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 9 Done Done Done false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 25 27 Done 25 25 1 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:11.305744139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:11.305913644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1392,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 63 63 63 104 104 104 T T F <--- 1 020-039 63 63 63 104 104 104 T T F <--- 2 040-059 61 61 61 99 99 99 T T F <--- 3 060-079 56 56 56 90 90 90 T T F <--- 4 080-099 61 61 61 99 99 99 F F F 5 100-119 64 64 64 105 105 105 T T F <--- 6 120-139 50 50 50 81 81 81 T T F <--- 7 140-159 64 64 64 105 105 105 T T F <--- 8 160-179 64 64 64 105 105 105 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 61 61 61 99 99 99 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 60 60 60 96 96 96 T T T 15 300-319 61 61 61 99 99 99 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 60 60 60 97 97 97 T T F <--- 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 63 63 63 104 104 104 T T F <--- 23 460-479 63 63 63 104 104 104 F F F 24 480-499 55 55 55 89 89 89 T T F <--- 25 500-519 59 59 59 94 94 94 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 62 62 62 103 103 103 F F F 29 580-599 62 62 62 103 103 103 F F F Max gen: 64, Max flush: 105 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 106 106 106 F F F 1 020-039 65 65 65 106 106 106 F F F 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 65 65 65 106 106 106 F F F 6 120-139 65 65 65 106 106 106 F F F 7 140-159 65 65 65 106 106 106 F F F 8 160-179 64 64 64 105 105 105 F F F 9 180-199 59 59 59 94 94 94 F F F 10 200-219 59 59 59 94 94 94 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 61 61 61 99 99 99 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 61 61 61 99 99 99 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 F F F 19 380-399 65 65 65 106 106 106 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 65 65 65 106 106 106 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 65 65 65 106 106 106 F F F 25 500-519 59 59 59 94 94 94 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 62 62 62 103 103 103 F F F 29 580-599 62 62 62 103 103 103 F F F Max gen: 65, Max flush: 106 Begin loop 33 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.559505892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.56776637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568083341Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.56811011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.56811866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568130199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Crucible 23c3c24a-4cf0-4fd1-a354-468e9e17f2e0 has session id: 0645a5fa-db99-4a49-a1d7-a29f6eff3c16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.5684693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Upstairs opts: Upstairs UUID: 23c3c24a-4cf0-4fd1-a354-468e9e17f2e0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568498649Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Crucible stats registered with UUID: 23c3c24a-4cf0-4fd1-a354-468e9e17f2e0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568512269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:15.568521648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568679584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568697833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568707833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568867999Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568883718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"23c3c24a-4cf0-4fd1-a354-468e9e17f2e0 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568920297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568929657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568939377Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568949146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568958716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568968756Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.568978595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"ds_connection connected from Ok(127.0.0.1:59390)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569018584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"ds_connection connected from Ok(127.0.0.1:40131)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569095002Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"ds_connection connected from Ok(127.0.0.1:58738)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569122561Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569674576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569708095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569824932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569974027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.569983537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[0]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 106, 106, 105, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570145923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[0]R generation[0..12]: [65, 65, 65, 65, 61, 65, 65, 65, 64, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570157192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570168282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[1]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 106, 106, 105, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570178932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[1]R generation[0..12]: [65, 65, 65, 65, 61, 65, 65, 65, 64, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570189361Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570200201Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[2]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 106, 106, 105, 94, 94, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570210831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[2]R generation[0..12]: [65, 65, 65, 65, 61, 65, 65, 65, 64, 59, 59, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.57022347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.57023457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.57025394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Next flush: 107","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570263349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570272869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"":"downstairs","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570282629Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570291529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"0","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570301268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"1","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570311018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"client":"2","":"downstairs","session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570320998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570330127Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"23c3c24a-4cf0-4fd1-a354-468e9e17f2e0 is now active with session: 0645a5fa-db99-4a49-a1d7-a29f6eff3c16","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570340187Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570348907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"session_id":"0645a5fa-db99-4a49-a1d7-a29f6eff3c16"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570358017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Activated sub_volume 23c3c24a-4cf0-4fd1-a354-468e9e17f2e0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570366016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570374276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.570736596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} 01/30 Flush 02/30 Flush 03/30 Write block 248 len 36864 data: 6 6 6 6 6 6 6 5 5 04/30 Read block 218 len 20480 05/30 Read block 97 len 32768 06/30 Read block 461 len 24576 07/30 Read block 459 len 12288 08/30 Write block 392 len 20480 data: 6 7 6 6 6 09/30 Flush 10/30 Write block 193 len 40960 data: 6 5 5 3 4 5 7 5 5 5 11/30 Write block 444 len 12288 data: 9 10 8 12/30 Read block 486 len 24576 13/30 Flush 14/30 Write block 171 len 8192 data: 3 4 15/30 Write block 306 len 32768 data: 6 5 5 4 4 4 4 4 16/30 Read block 571 len 12288 17/30 Write block 121 len 40960 data: 8 8 9 7 8 8 8 6 4 4 18/30 Write block 584 len 32768 data: 7 6 5 5 7 5 4 4 19/30 Write block 564 len 28672 data: 7 6 6 7 7 5 6 20/30 Read block 461 len 28672 21/30 Read block 577 len 28672 22/30 Write block 512 len 4096 data: 4 23/30 Read block 526 len 16384 24/30 Write block 497 len 36864 data: 5 5 3 2 3 4 6 6 5 25/30 Write block 363 len 12288 data: 4 4 6 26/30 Read block 77 len 12288 27/30 Write block 161 len 8192 data: 6 7 28/30 Read block 518 len 8192 29/30 Read block 364 len 40960 30/30 Write block 250 len 36864 data: 7 7 7 7 7 6 6 4 3 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 3 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1012)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:15.579734253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:15.579914728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1400,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 106 106 106 F F F 1 020-039 65 65 65 106 106 106 F F F 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 65 65 65 106 106 106 F F F 6 120-139 65 65 65 106 106 106 T T F <--- 7 140-159 65 65 65 106 106 106 F F F 8 160-179 64 64 64 105 105 105 T T F <--- 9 180-199 66 66 59 110 110 94 F F F <--- 10 200-219 66 66 59 110 110 94 F F F <--- 11 220-239 54 54 54 88 88 88 F F F 12 240-259 66 66 61 109 109 99 T T F <--- 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 61 61 61 99 99 99 T T F <--- 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 59 59 59 94 94 94 T T F <--- 19 380-399 66 66 65 109 109 106 F F F <--- 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 65 110 110 106 F F F <--- 23 460-479 63 63 63 104 104 104 F F F 24 480-499 65 65 65 106 106 106 T T F <--- 25 500-519 59 59 59 94 94 94 T T F <--- 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 62 62 62 103 103 103 T T F <--- 29 580-599 62 62 62 103 103 103 T T F <--- Max gen: 66, Max flush: 110 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 65 65 65 106 106 106 F F F 1 020-039 65 65 65 106 106 106 F F F 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 61 61 61 99 99 99 F F F 5 100-119 65 65 65 106 106 106 F F F 6 120-139 67 67 67 111 111 111 F F F 7 140-159 65 65 65 106 106 106 F F F 8 160-179 67 67 67 111 111 111 F F F 9 180-199 66 66 66 110 110 110 F F F 10 200-219 66 66 66 110 110 110 F F F 11 220-239 54 54 54 88 88 88 F F F 12 240-259 67 67 67 111 111 111 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 61 61 61 99 99 99 F F F 18 360-379 67 67 67 111 111 111 F F F 19 380-399 66 66 66 109 109 109 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 67 67 67 111 111 111 F F F 25 500-519 67 67 67 111 111 111 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 67 67 67 111 111 111 F F F 29 580-599 67 67 67 111 111 111 F F F Max gen: 67, Max flush: 111 Begin loop 34 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.8343998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84257469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.842857902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.842887661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.842896401Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.842907951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Crucible cac95aa3-906d-458a-9a82-4410d7d07e71 has session id: c3f35b41-9af2-4123-be15-85850e3ce55a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843244381Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Upstairs opts: Upstairs UUID: cac95aa3-906d-458a-9a82-4410d7d07e71, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84327409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Crucible stats registered with UUID: cac95aa3-906d-458a-9a82-4410d7d07e71","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84328556Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:19.84329495Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843419256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843437306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843447615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843586191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843604191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"cac95aa3-906d-458a-9a82-4410d7d07e71 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84362761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84363693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84364664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843656459Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843666199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843676239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843686109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"ds_connection connected from Ok(127.0.0.1:48269)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843803435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"ds_connection connected from Ok(127.0.0.1:33512)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843872503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46237)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.843884683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844463787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844489736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844501886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844769028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844788998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[0]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 111, 106, 111, 110, 110, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844851826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[0]R generation[0..12]: [65, 65, 65, 65, 61, 65, 67, 65, 67, 66, 66, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844863046Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844874185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[1]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 111, 106, 111, 110, 110, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844884925Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[1]R generation[0..12]: [65, 65, 65, 65, 61, 65, 67, 65, 67, 66, 66, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844895265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844906284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[2]R flush_numbers[0..12]: [106, 106, 106, 106, 99, 106, 111, 106, 111, 110, 110, 88]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844916814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[2]R generation[0..12]: [65, 65, 65, 65, 61, 65, 67, 65, 67, 66, 66, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844927254Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844938293Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844947763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844957503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Next flush: 112","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844966813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844976232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"":"downstairs","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844987812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.844996672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"0","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.845006322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"1","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.845015991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"client":"2","":"downstairs","session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.845025791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.845034821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"cac95aa3-906d-458a-9a82-4410d7d07e71 is now active with session: c3f35b41-9af2-4123-be15-85850e3ce55a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84504484Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84505371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"session_id":"c3f35b41-9af2-4123-be15-85850e3ce55a"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84506305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Activated sub_volume cac95aa3-906d-458a-9a82-4410d7d07e71","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84507111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84507913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.84541937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} 01/30 Write block 364 len 24576 data: 5 7 5 5 5 6 02/30 Write block 511 len 4096 data: 4 03/30 Write block 93 len 16384 data: 3 4 5 5 04/30 Write block 390 len 32768 data: 6 4 7 8 7 7 7 5 05/30 Read block 120 len 36864 06/30 Read block 43 len 28672 07/30 Write block 14 len 12288 data: 8 7 7 08/30 Write block 250 len 24576 data: 8 8 8 8 8 7 09/30 Write block 340 len 28672 data: 5 5 4 4 4 4 5 10/30 Read block 191 len 12288 11/30 Write block 75 len 12288 data: 5 6 6 12/30 Write block 583 len 40960 data: 8 8 7 6 6 8 6 5 5 5 13/30 Flush 14/30 Write block 178 len 40960 data: 2 3 4 5 6 7 8 8 9 10 15/30 Read block 500 len 32768 16/30 Flush 17/30 Write block 162 len 28672 data: 8 8 8 8 7 7 8 18/30 Write block 59 len 4096 data: 8 19/30 Read block 318 len 32768 20/30 Read block 205 len 12288 21/30 Write block 160 len 8192 data: 5 7 22/30 Write block 107 len 28672 data: 3 5 6 5 6 5 6 23/30 Read block 172 len 28672 24/30 Write block 576 len 12288 data: 6 8 9 25/30 Read block 401 len 32768 26/30 Write block 170 len 36864 data: 4 4 5 4 5 5 4 3 3 27/30 Read block 232 len 20480 28/30 Write block 585 len 12288 data: 8 7 7 29/30 Read block 168 len 32768 30/30 Write block 231 len 4096 data: 4 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Write 6 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 25 1 27 Done 29 5 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:19.853711247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:19.853891972Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1408,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 65 68 112 106 112 F F F <--- 1 020-039 65 65 65 106 106 106 F F F 2 040-059 65 65 65 106 106 106 T F T <--- 3 060-079 68 65 68 112 106 112 F F F <--- 4 080-099 68 61 68 112 99 112 F T F <--- 5 100-119 65 65 65 106 106 106 T F T <--- 6 120-139 67 67 67 111 111 111 F F F 7 140-159 65 65 65 106 106 106 F F F 8 160-179 68 67 68 113 111 113 T F T <--- 9 180-199 68 66 68 113 110 113 F F F <--- 10 200-219 66 66 66 110 110 110 F F F 11 220-239 54 54 54 88 88 88 T F T <--- 12 240-259 68 67 68 112 111 112 F F F <--- 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 68 61 68 112 99 112 F F F <--- 18 360-379 68 67 68 112 111 112 F T F <--- 19 380-399 68 66 68 112 109 112 F T F <--- 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 67 67 67 111 111 111 F F F 25 500-519 68 67 68 112 111 112 F T F <--- 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 67 67 67 111 111 111 T F T <--- 29 580-599 68 67 68 112 111 112 T F T <--- Max gen: 68, Max flush: 113 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 112 112 112 F F F 1 020-039 65 65 65 106 106 106 F F F 2 040-059 69 69 69 114 114 114 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 68 68 68 112 112 112 F F F 5 100-119 69 69 69 114 114 114 F F F 6 120-139 67 67 67 111 111 111 F F F 7 140-159 65 65 65 106 106 106 F F F 8 160-179 69 69 69 114 114 114 F F F 9 180-199 68 68 68 113 113 113 F F F 10 200-219 66 66 66 110 110 110 F F F 11 220-239 69 69 69 114 114 114 F F F 12 240-259 68 68 68 112 112 112 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 68 68 68 112 112 112 F F F 18 360-379 68 68 68 112 112 112 F F F 19 380-399 68 68 68 112 112 112 F F F 20 400-419 61 61 61 99 99 99 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 67 67 67 111 111 111 F F F 25 500-519 68 68 68 112 112 112 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 69, Max flush: 114 Begin loop 35 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.113017494Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.121561553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.121850695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.121876335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.121887694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.121899584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Crucible d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2 has session id: 1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122262444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Upstairs opts: Upstairs UUID: d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122290963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Crucible stats registered with UUID: d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122301813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:24.122310952Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122466098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122482898Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122493057Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122625464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122640783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122675342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122684842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122694592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122706351Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122716231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122726201Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.12273632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:41124)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122775559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:51201)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122896206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:55510)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.122920645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123485109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123512439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123548828Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123780811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.12380547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[0]R flush_numbers[0..12]: [112, 106, 114, 112, 112, 114, 111, 106, 114, 113, 110, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123852029Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[0]R generation[0..12]: [68, 65, 69, 68, 68, 69, 67, 65, 69, 68, 66, 69]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123863159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123874328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[1]R flush_numbers[0..12]: [112, 106, 114, 112, 112, 114, 111, 106, 114, 113, 110, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123887818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[1]R generation[0..12]: [68, 65, 69, 68, 68, 69, 67, 65, 69, 68, 66, 69]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123898258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123909387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[2]R flush_numbers[0..12]: [112, 106, 114, 112, 112, 114, 111, 106, 114, 113, 110, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123920007Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[2]R generation[0..12]: [68, 65, 69, 68, 68, 69, 67, 65, 69, 68, 66, 69]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123930457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123941537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123951066Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123960816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Next flush: 115","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123970216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.123979746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"":"downstairs","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124011035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124020014Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"0","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124029714Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"1","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124039474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"client":"2","":"downstairs","session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124049264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124059783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2 is now active with session: 1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124069753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124078853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124086523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"session_id":"1ab5b8f7-cbac-4d5d-be12-54b4e2def3aa"} {"msg":"Activated sub_volume d2a2ef7d-d145-40b2-be85-a7ffc5dfc5f2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124095702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124103672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.124465932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} 01/30 Write block 380 len 40960 data: 5 6 6 7 8 8 7 8 8 8 02/30 Read block 376 len 36864 03/30 Read block 350 len 28672 04/30 Read block 376 len 16384 05/30 Write block 177 len 16384 data: 4 4 4 5 06/30 Read block 462 len 8192 07/30 Read block 535 len 36864 08/30 Write block 134 len 24576 data: 3 3 4 5 5 5 09/30 Read block 159 len 4096 10/30 Read block 461 len 40960 11/30 Read block 261 len 28672 12/30 Read block 123 len 8192 13/30 Write block 201 len 24576 data: 6 6 6 5 6 4 14/30 Read block 508 len 28672 15/30 Write block 39 len 32768 data: 8 10 9 11 11 10 9 9 16/30 Write block 196 len 20480 data: 4 5 6 8 6 17/30 Write block 500 len 8192 data: 3 4 18/30 Write block 137 len 40960 data: 6 6 6 5 6 5 5 6 6 5 19/30 Read block 596 len 8192 20/30 Read block 318 len 20480 21/30 Read block 151 len 16384 22/30 Read block 335 len 4096 23/30 Flush 24/30 Write block 238 len 12288 data: 6 7 7 25/30 Write block 400 len 32768 data: 6 6 6 5 5 5 5 5 26/30 Read block 47 len 36864 27/30 Read block 177 len 8192 28/30 Write block 208 len 12288 data: 3 6 5 29/30 Write block 46 len 8192 data: 10 13 30/30 Read block 140 len 24576 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:24.133341042Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:24.133514917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1416,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 112 112 112 F F F 1 020-039 65 70 70 106 115 115 F F F <--- 2 040-059 69 70 70 114 115 115 F T T <--- 3 060-079 68 68 68 112 112 112 F F F 4 080-099 68 68 68 112 112 112 F F F 5 100-119 69 69 69 114 114 114 F F F 6 120-139 67 70 70 111 115 115 F F F <--- 7 140-159 65 70 70 106 115 115 F F F <--- 8 160-179 69 70 70 114 115 115 F F F <--- 9 180-199 68 70 70 113 115 115 F F F <--- 10 200-219 66 70 70 110 115 115 F T T <--- 11 220-239 69 69 69 114 114 114 F T T <--- 12 240-259 68 68 68 112 112 112 F T T <--- 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 68 68 68 112 112 112 F F F 18 360-379 68 68 68 112 112 112 F F F 19 380-399 68 70 70 112 115 115 T F F <--- 20 400-419 61 61 61 99 99 99 F T T <--- 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 67 67 67 111 111 111 F F F 25 500-519 68 70 70 112 115 115 F F F <--- 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 70, Max flush: 115 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 112 112 112 F F F 1 020-039 70 70 70 115 115 115 F F F 2 040-059 71 71 71 116 116 116 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 68 68 68 112 112 112 F F F 5 100-119 69 69 69 114 114 114 F F F 6 120-139 70 70 70 115 115 115 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 70 70 70 115 115 115 F F F 10 200-219 71 71 71 116 116 116 F F F 11 220-239 71 71 71 116 116 116 F F F 12 240-259 71 71 71 116 116 116 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 68 68 68 112 112 112 F F F 18 360-379 68 68 68 112 112 112 F F F 19 380-399 70 70 70 115 115 115 F F F 20 400-419 71 71 71 116 116 116 F F F 21 420-439 63 63 63 104 104 104 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 67 67 67 111 111 111 F F F 25 500-519 70 70 70 115 115 115 F F F 26 520-539 63 63 63 104 104 104 F F F 27 540-559 61 61 61 99 99 99 F F F 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 71, Max flush: 116 Begin loop 36 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.391228499Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.399522716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.399810078Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.399840957Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.399849657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.399861386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Crucible f8512bac-84f7-481a-90a5-8aeed94c124a has session id: f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400196987Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Upstairs opts: Upstairs UUID: f8512bac-84f7-481a-90a5-8aeed94c124a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400227026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Crucible stats registered with UUID: f8512bac-84f7-481a-90a5-8aeed94c124a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400240486Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:28.400249815Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400381902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400399221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400409381Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400565567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400579816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"f8512bac-84f7-481a-90a5-8aeed94c124a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400603556Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400613045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400622855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400632575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400642364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400652434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400662474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"ds_connection connected from Ok(127.0.0.1:60982)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400731452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"ds_connection connected from Ok(127.0.0.1:58183)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400756931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"ds_connection connected from Ok(127.0.0.1:39527)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.400770531Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401391533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.40149766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.40152845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401746603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401770363Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[0]R flush_numbers[0..12]: [112, 115, 116, 112, 112, 114, 115, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.4018668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[0]R generation[0..12]: [68, 70, 71, 68, 68, 69, 70, 70, 70, 70, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.40187888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401890489Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[1]R flush_numbers[0..12]: [112, 115, 116, 112, 112, 114, 115, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401901379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[1]R generation[0..12]: [68, 70, 71, 68, 68, 69, 70, 70, 70, 70, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401911849Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401923008Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[2]R flush_numbers[0..12]: [112, 115, 116, 112, 112, 114, 115, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401933758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[2]R generation[0..12]: [68, 70, 71, 68, 68, 69, 70, 70, 70, 70, 71, 71]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401944208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401958137Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401967777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401977517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Next flush: 117","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401986987Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.401996516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"":"downstairs","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402006346Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402015196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"0","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402024956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"1","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402034725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"client":"2","":"downstairs","session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402044665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402053785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"f8512bac-84f7-481a-90a5-8aeed94c124a is now active with session: f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402063884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402072654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"session_id":"f5f9ec98-2f2f-4d2e-9aff-7a0f4e64d862"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402081654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Activated sub_volume f8512bac-84f7-481a-90a5-8aeed94c124a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402089664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402097824Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.402395845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} 01/30 Write block 512 len 28672 data: 5 6 7 6 7 7 8 02/30 Read block 117 len 8192 03/30 Write block 277 len 8192 data: 9 8 04/30 Read block 457 len 12288 05/30 Read block 5 len 28672 06/30 Read block 279 len 16384 07/30 Read block 93 len 12288 08/30 Write block 417 len 16384 data: 5 4 4 6 09/30 Write block 117 len 20480 data: 7 8 10 7 9 10/30 Write block 495 len 16384 data: 5 5 6 6 11/30 Write block 18 len 8192 data: 7 9 12/30 Write block 84 len 20480 data: 5 4 4 6 7 13/30 Write block 352 len 16384 data: 9 8 6 5 14/30 Flush 15/30 Flush 16/30 Write block 101 len 36864 data: 6 6 4 5 3 3 4 6 7 17/30 Flush 18/30 Write block 8 len 4096 data: 7 19/30 Write block 219 len 32768 data: 4 6 8 6 6 6 6 6 20/30 Write block 118 len 20480 data: 9 11 8 10 9 21/30 Flush 22/30 Write block 550 len 32768 data: 5 5 7 7 7 8 9 8 23/30 Write block 518 len 12288 data: 9 10 10 24/30 Read block 149 len 8192 25/30 Flush 26/30 Flush 27/30 Write block 425 len 28672 data: 8 7 5 5 6 7 9 28/30 Read block 465 len 8192 29/30 Read block 235 len 28672 30/30 Write block 0 len 40960 data: 3 3 4 4 6 5 6 8 8 5 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 7 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 23 1 25 Done 29 7 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:28.410499427Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:28.410703952Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1424,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 68 72 120 112 120 T F T <--- 1 020-039 70 70 70 115 115 115 F F F 2 040-059 71 71 71 116 116 116 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 68 72 117 112 117 F F F <--- 5 100-119 72 69 72 120 114 120 F F F <--- 6 120-139 72 70 72 120 115 120 F F F <--- 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 70 70 70 115 115 115 F F F 10 200-219 72 71 72 120 116 120 F F F <--- 11 220-239 72 71 72 120 116 120 F F F <--- 12 240-259 71 71 71 116 116 116 F F F 13 260-279 72 63 72 117 104 117 F T F <--- 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 72 68 72 117 112 117 F F F <--- 18 360-379 68 68 68 112 112 112 F F F 19 380-399 70 70 70 115 115 115 F F F 20 400-419 72 71 72 117 116 117 F F F <--- 21 420-439 72 63 72 117 104 117 T F T <--- 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 72 67 72 117 111 117 F F F <--- 25 500-519 72 70 72 121 115 121 F T F <--- 26 520-539 72 63 72 121 104 121 F F F <--- 27 540-559 72 61 72 121 99 121 F F F <--- 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 72, Max flush: 121 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 70 70 70 115 115 115 F F F 2 040-059 71 71 71 116 116 116 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 72 72 117 117 117 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 70 70 70 115 115 115 F F F 10 200-219 72 72 72 120 120 120 F F F 11 220-239 72 72 72 120 120 120 F F F 12 240-259 71 71 71 116 116 116 F F F 13 260-279 72 72 72 117 117 117 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 61 61 99 99 99 F F F 17 340-359 72 72 72 117 117 117 F F F 18 360-379 68 68 68 112 112 112 F F F 19 380-399 70 70 70 115 115 115 F F F 20 400-419 72 72 72 117 117 117 F F F 21 420-439 73 73 73 122 122 122 F F F 22 440-459 66 66 66 110 110 110 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 72 72 72 117 117 117 F F F 25 500-519 72 72 72 121 121 121 F F F 26 520-539 72 72 72 121 121 121 F F F 27 540-559 72 72 72 121 121 121 F F F 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 73, Max flush: 122 Begin loop 37 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.66887812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:32.67600647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.676063098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683037212Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683296425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683320594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683329364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683341034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Crucible d8f0ab68-59b4-42de-82e4-87e24d3120da has session id: 5d4a3120-7bea-481e-85a8-b20ffca150e5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683698694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Upstairs opts: Upstairs UUID: d8f0ab68-59b4-42de-82e4-87e24d3120da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683721313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Crucible stats registered with UUID: d8f0ab68-59b4-42de-82e4-87e24d3120da","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683731583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:32.683740892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683893918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683912268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.683922367Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684059823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684073323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"d8f0ab68-59b4-42de-82e4-87e24d3120da active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684113352Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684122762Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684132401Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684142111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684151661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684161511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684243418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:34686)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684268748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:38731)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684280447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:35340)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.684291097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685001157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685030206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685096214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685292399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685308618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[0]R flush_numbers[0..12]: [122, 115, 116, 112, 117, 120, 120, 115, 115, 115, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685393356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[0]R generation[0..12]: [73, 70, 71, 68, 72, 72, 72, 70, 70, 70, 72, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685404606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685415665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[1]R flush_numbers[0..12]: [122, 115, 116, 112, 117, 120, 120, 115, 115, 115, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685426335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[1]R generation[0..12]: [73, 70, 71, 68, 72, 72, 72, 70, 70, 70, 72, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685436655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685447594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[2]R flush_numbers[0..12]: [122, 115, 116, 112, 117, 120, 120, 115, 115, 115, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685458204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[2]R generation[0..12]: [73, 70, 71, 68, 72, 72, 72, 70, 70, 70, 72, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685468444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685479324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685488693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685498273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Next flush: 123","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685509173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685518612Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"":"downstairs","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685528592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685537472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"0","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685547332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"1","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685557101Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"client":"2","":"downstairs","session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685567311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685576501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"d8f0ab68-59b4-42de-82e4-87e24d3120da is now active with session: 5d4a3120-7bea-481e-85a8-b20ffca150e5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685586351Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.68559492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"session_id":"5d4a3120-7bea-481e-85a8-b20ffca150e5"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.6856043Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Activated sub_volume d8f0ab68-59b4-42de-82e4-87e24d3120da","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.68561245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.68562074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.685985169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} 01/30 Write block 404 len 28672 data: 6 6 6 6 6 6 5 02/30 Read block 16 len 20480 03/30 Flush 04/30 Write block 246 len 24576 data: 5 5 7 7 9 9 05/30 Read block 155 len 16384 06/30 Read block 307 len 16384 07/30 Read block 581 len 36864 08/30 Write block 457 len 12288 data: 11 9 7 09/30 Read block 244 len 36864 10/30 Read block 392 len 4096 11/30 Read block 375 len 16384 12/30 Write block 198 len 24576 data: 7 9 7 7 7 7 13/30 Read block 320 len 12288 14/30 Write block 43 len 32768 data: 12 11 10 11 14 10 10 10 15/30 Write block 246 len 36864 data: 6 6 8 8 10 10 9 9 9 16/30 Write block 46 len 8192 data: 12 15 17/30 Read block 73 len 32768 18/30 Read block 198 len 40960 19/30 Read block 458 len 16384 20/30 Read block 316 len 16384 21/30 Flush 22/30 Write block 327 len 40960 data: 9 8 10 8 6 6 5 5 7 6 23/30 Read block 572 len 36864 24/30 Flush 25/30 Write block 395 len 16384 data: 8 8 6 5 26/30 Read block 286 len 24576 27/30 Write block 368 len 40960 data: 6 7 5 6 5 5 6 7 7 8 28/30 Write block 423 len 28672 data: 7 7 9 8 6 6 7 29/30 Write block 258 len 32768 data: 4 5 5 5 5 4 6 7 30/30 Write block 537 len 28672 data: 8 7 5 6 6 7 8 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 6 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 3 3 33 Done 0 24 24 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:32.69522479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:32.695403375Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1432,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 70 70 70 115 115 115 F F F 2 040-059 71 74 74 116 124 124 F F F <--- 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 72 72 117 117 117 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 70 74 74 115 124 124 F F F <--- 10 200-219 72 74 74 120 124 124 F F F <--- 11 220-239 72 72 72 120 120 120 F F F 12 240-259 71 74 74 116 124 124 F T T <--- 13 260-279 72 72 72 117 117 117 F T T <--- 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 61 74 74 99 125 125 F F F <--- 17 340-359 72 72 72 117 117 117 F F F 18 360-379 68 68 68 112 112 112 F T T <--- 19 380-399 70 70 70 115 115 115 F T T <--- 20 400-419 74 74 74 123 123 123 F F F 21 420-439 73 73 73 122 122 122 F T T <--- 22 440-459 66 74 74 110 124 124 F F F <--- 23 460-479 63 63 63 104 104 104 F F F 24 480-499 72 72 72 117 117 117 F F F 25 500-519 72 72 72 121 121 121 F F F 26 520-539 72 72 72 121 121 121 F T T <--- 27 540-559 72 72 72 121 121 121 F T T <--- 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 74, Max flush: 125 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 70 70 70 115 115 115 F F F 2 040-059 74 74 74 124 124 124 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 72 72 117 117 117 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 74 74 74 124 124 124 F F F 10 200-219 74 74 74 124 124 124 F F F 11 220-239 72 72 72 120 120 120 F F F 12 240-259 75 75 75 126 126 126 F F F 13 260-279 75 75 75 126 126 126 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 67 67 111 111 111 F F F 16 320-339 74 74 74 125 125 125 F F F 17 340-359 72 72 72 117 117 117 F F F 18 360-379 75 75 75 126 126 126 F F F 19 380-399 75 75 75 126 126 126 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 75 75 75 126 126 126 F F F 22 440-459 74 74 74 124 124 124 F F F 23 460-479 63 63 63 104 104 104 F F F 24 480-499 72 72 72 117 117 117 F F F 25 500-519 72 72 72 121 121 121 F F F 26 520-539 75 75 75 126 126 126 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 69 69 69 114 114 114 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 75, Max flush: 126 Begin loop 38 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.949498167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:36.956655316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.956712264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.963561252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.963827774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.963854154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.963862553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.963874123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Crucible dc0e2489-52f2-47b4-b5d1-da379c924896 has session id: 7f647f7d-3eb9-4e29-992c-accf22b555ea","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964183064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Upstairs opts: Upstairs UUID: dc0e2489-52f2-47b4-b5d1-da379c924896, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964206554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Crucible stats registered with UUID: dc0e2489-52f2-47b4-b5d1-da379c924896","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964216414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:36.964225433Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964371249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964388399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964398468Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964526915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964537364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"dc0e2489-52f2-47b4-b5d1-da379c924896 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964553824Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964564404Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964574033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964583733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964593383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964604353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964649371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:45347)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964660341Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:48181)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.96470138Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:51784)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.964722509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965366541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965379141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965431019Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965681132Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965706072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[0]R flush_numbers[0..12]: [122, 115, 124, 112, 117, 120, 120, 115, 115, 124, 124, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.96574709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[0]R generation[0..12]: [73, 70, 74, 68, 72, 72, 72, 70, 70, 74, 74, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.96576091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.96577222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[1]R flush_numbers[0..12]: [122, 115, 124, 112, 117, 120, 120, 115, 115, 124, 124, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965782889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[1]R generation[0..12]: [73, 70, 74, 68, 72, 72, 72, 70, 70, 74, 74, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965793419Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965804259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[2]R flush_numbers[0..12]: [122, 115, 124, 112, 117, 120, 120, 115, 115, 124, 124, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965814919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[2]R generation[0..12]: [73, 70, 74, 68, 72, 72, 72, 70, 70, 74, 74, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965825268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965836298Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965845678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965855507Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Next flush: 127","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965864987Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965874517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"":"downstairs","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965884357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965893256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"0","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965904636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"1","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965914466Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"client":"2","":"downstairs","session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965924456Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965933675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"dc0e2489-52f2-47b4-b5d1-da379c924896 is now active with session: 7f647f7d-3eb9-4e29-992c-accf22b555ea","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965943625Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965952315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"session_id":"7f647f7d-3eb9-4e29-992c-accf22b555ea"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965961394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Activated sub_volume dc0e2489-52f2-47b4-b5d1-da379c924896","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965969484Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.965977614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.966311655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} 01/30 Write block 497 len 28672 data: 7 7 4 4 5 5 7 02/30 Read block 175 len 16384 03/30 Read block 366 len 12288 04/30 Write block 395 len 16384 data: 9 9 7 6 05/30 Read block 147 len 8192 06/30 Read block 458 len 4096 07/30 Read block 579 len 40960 08/30 Write block 529 len 36864 data: 5 6 7 7 5 5 5 7 9 09/30 Write block 492 len 20480 data: 6 6 5 6 6 10/30 Read block 548 len 24576 11/30 Write block 248 len 28672 data: 9 9 11 11 10 10 10 12/30 Write block 203 len 8192 data: 8 6 13/30 Write block 309 len 40960 data: 5 5 5 5 5 4 4 3 5 5 14/30 Write block 300 len 20480 data: 6 5 4 4 4 15/30 Write block 265 len 40960 data: 8 6 6 6 8 10 7 7 7 7 16/30 Read block 31 len 24576 17/30 Write block 209 len 12288 data: 7 6 5 18/30 Write block 29 len 8192 data: 5 6 19/30 Read block 3 len 40960 20/30 Write block 562 len 16384 data: 8 6 8 7 21/30 Read block 62 len 36864 22/30 Write block 315 len 8192 data: 5 4 23/30 Write block 464 len 4096 data: 5 24/30 Read block 569 len 36864 25/30 Write block 316 len 16384 data: 5 6 6 6 26/30 Flush 27/30 Write block 482 len 28672 data: 4 4 4 4 3 4 5 28/30 Write block 25 len 28672 data: 8 8 7 6 6 7 7 29/30 Read block 103 len 8192 30/30 Read block 527 len 36864 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 4 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 9 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 0 0 25 Done 5 30 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:36.974685249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:36.974852365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1440,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 70 76 76 115 127 127 F T T <--- 2 040-059 74 74 74 124 124 124 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 72 72 117 117 117 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 74 74 74 124 124 124 F F F 10 200-219 74 76 76 124 127 127 F F F <--- 11 220-239 72 72 72 120 120 120 F F F 12 240-259 75 76 76 126 127 127 F F F <--- 13 260-279 75 76 76 126 127 127 F F F <--- 14 280-299 65 65 65 106 106 106 F F F 15 300-319 67 76 76 111 127 127 F F F <--- 16 320-339 74 74 74 125 125 125 F F F 17 340-359 72 72 72 117 117 117 F F F 18 360-379 75 75 75 126 126 126 F F F 19 380-399 75 76 76 126 127 127 T F F <--- 20 400-419 74 74 74 123 123 123 F F F 21 420-439 75 75 75 126 126 126 F F F 22 440-459 74 74 74 124 124 124 F F F 23 460-479 63 76 76 104 127 127 F F F <--- 24 480-499 72 76 76 117 127 127 T T T <--- 25 500-519 72 76 76 121 127 127 T F F <--- 26 520-539 75 76 76 126 127 127 F F F <--- 27 540-559 75 75 75 126 126 126 F F F 28 560-579 69 76 76 114 127 127 F F F <--- 29 580-599 69 69 69 114 114 114 F F F Max gen: 76, Max flush: 127 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 74 74 74 124 124 124 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 72 72 72 117 117 117 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 70 70 70 115 115 115 F F F 8 160-179 70 70 70 115 115 115 F F F 9 180-199 74 74 74 124 124 124 F F F 10 200-219 76 76 76 127 127 127 F F F 11 220-239 72 72 72 120 120 120 F F F 12 240-259 76 76 76 127 127 127 F F F 13 260-279 76 76 76 127 127 127 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 76 76 76 127 127 127 F F F 16 320-339 74 74 74 125 125 125 F F F 17 340-359 72 72 72 117 117 117 F F F 18 360-379 75 75 75 126 126 126 F F F 19 380-399 76 76 76 127 127 127 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 75 75 75 126 126 126 F F F 22 440-459 74 74 74 124 124 124 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 77 77 77 128 128 128 F F F 25 500-519 76 76 76 127 127 127 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 77, Max flush: 128 Begin loop 39 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.247205595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.255505451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.255791413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.255817153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.255825832Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.255837592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Crucible ffb1d419-a3b8-4dc4-88a6-2c0f473feeef has session id: 9ad5e7d7-082a-462a-a230-19630f85fe0c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256145753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Upstairs opts: Upstairs UUID: ffb1d419-a3b8-4dc4-88a6-2c0f473feeef, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256168843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Crucible stats registered with UUID: ffb1d419-a3b8-4dc4-88a6-2c0f473feeef","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256178982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:41.256188072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256346718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256362857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256372987Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256492164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256508813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"ffb1d419-a3b8-4dc4-88a6-2c0f473feeef active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256540992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256550282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256560182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256569871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256579561Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256589761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.25662904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38900)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256653899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:62604)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256665729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:39716)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.256676318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257307431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.25733273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.25734492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257641371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257651091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[0]R flush_numbers[0..12]: [122, 128, 124, 112, 117, 120, 120, 115, 115, 124, 127, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.25768972Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[0]R generation[0..12]: [73, 77, 74, 68, 72, 72, 72, 70, 70, 74, 76, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.25770079Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257712089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[1]R flush_numbers[0..12]: [122, 128, 124, 112, 117, 120, 120, 115, 115, 124, 127, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257722879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[1]R generation[0..12]: [73, 77, 74, 68, 72, 72, 72, 70, 70, 74, 76, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257733469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257747058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[2]R flush_numbers[0..12]: [122, 128, 124, 112, 117, 120, 120, 115, 115, 124, 127, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257757888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[2]R generation[0..12]: [73, 77, 74, 68, 72, 72, 72, 70, 70, 74, 76, 72]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257768398Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257779427Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257788917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257798697Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Next flush: 129","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257808337Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257817806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"":"downstairs","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257827576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257836416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"0","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257846365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"1","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257856285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"client":"2","":"downstairs","session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257866125Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257875345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"ffb1d419-a3b8-4dc4-88a6-2c0f473feeef is now active with session: 9ad5e7d7-082a-462a-a230-19630f85fe0c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257885294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257895684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"session_id":"9ad5e7d7-082a-462a-a230-19630f85fe0c"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257904784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Activated sub_volume ffb1d419-a3b8-4dc4-88a6-2c0f473feeef","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.257913204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"timeDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"2026-09-24T22:11:41.257921483Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.258207415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} 01/30 Write block 195 len 40960 data: 6 5 6 8 10 8 8 8 9 7 02/30 Write block 426 len 4096 data: 9 03/30 Write block 359 len 20480 data: 7 5 5 4 5 04/30 Write block 304 len 20480 data: 5 4 7 6 6 05/30 Read block 312 len 4096 06/30 Read block 52 len 4096 07/30 Write block 149 len 20480 data: 8 7 8 9 9 08/30 Flush 09/30 Flush 10/30 Write block 482 len 12288 data: 5 5 5 11/30 Read block 270 len 32768 12/30 Write block 177 len 12288 data: 5 5 5 13/30 Write block 447 len 4096 data: 6 14/30 Write block 50 len 28672 data: 11 9 7 8 7 6 7 15/30 Write block 225 len 32768 data: 7 7 6 5 6 6 5 4 16/30 Write block 151 len 8192 data: 9 10 17/30 Read block 452 len 24576 18/30 Write block 225 len 28672 data: 8 8 7 6 7 7 6 19/30 Write block 300 len 12288 data: 7 6 5 20/30 Write block 193 len 16384 data: 7 6 7 6 21/30 Read block 481 len 40960 22/30 Flush 23/30 Write block 81 len 8192 data: 5 5 24/30 Read block 519 len 8192 25/30 Write block 233 len 16384 data: 5 7 7 5 26/30 Flush 27/30 Write block 208 len 36864 data: 4 8 7 6 7 5 4 4 5 28/30 Read block 421 len 36864 29/30 Read block 64 len 20480 30/30 Read block 31 len 32768 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Read 1 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 25 26 Done 30 29 5 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:41.266185041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:41.266359586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1448,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 78 78 74 131 131 124 F F F <--- 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 72 132 132 117 F F F <--- 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 78 78 70 131 131 115 F F F <--- 8 160-179 78 78 70 131 131 115 F F F <--- 9 180-199 78 78 74 131 131 124 F F T <--- 10 200-219 78 78 76 129 129 127 T T T <--- 11 220-239 78 78 72 132 132 120 F F F <--- 12 240-259 76 76 76 127 127 127 F F F 13 260-279 76 76 76 127 127 127 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 78 78 76 131 131 127 F F T <--- 16 320-339 74 74 74 125 125 125 F F F 17 340-359 78 78 72 129 129 117 F F T <--- 18 360-379 78 78 75 129 129 126 F F T <--- 19 380-399 76 76 76 127 127 127 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 78 78 75 129 129 126 F F T <--- 22 440-459 78 78 74 131 131 124 F F F <--- 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 77 131 131 128 F F F <--- 25 500-519 76 76 76 127 127 127 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 78, Max flush: 132 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 78 78 78 131 131 131 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 78 132 132 132 F F F 5 100-119 72 72 72 120 120 120 F F F 6 120-139 72 72 72 120 120 120 F F F 7 140-159 78 78 78 131 131 131 F F F 8 160-179 78 78 78 131 131 131 F F F 9 180-199 78 78 78 131 131 131 F F F 10 200-219 79 79 79 133 133 133 F F F 11 220-239 78 78 78 132 132 132 F F F 12 240-259 76 76 76 127 127 127 F F F 13 260-279 76 76 76 127 127 127 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 78 78 78 131 131 131 F F F 16 320-339 74 74 74 125 125 125 F F F 17 340-359 78 78 78 129 129 129 F F F 18 360-379 78 78 78 129 129 129 F F F 19 380-399 76 76 76 127 127 127 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 78 78 78 129 129 129 F F F 22 440-459 78 78 78 131 131 131 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F F F 25 500-519 76 76 76 127 127 127 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 79, Max flush: 133 Begin loop 40 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.521345554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.529539414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.529867735Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.529892164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.529900944Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.529912784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Crucible 64f4f2ec-7112-49b8-a86f-8acd428f967e has session id: b74096f1-b786-470c-abdc-e938ac4e5f8d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530257034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Upstairs opts: Upstairs UUID: 64f4f2ec-7112-49b8-a86f-8acd428f967e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530283503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Crucible stats registered with UUID: 64f4f2ec-7112-49b8-a86f-8acd428f967e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530293993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:45.530303153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530462928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530491257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530502077Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530617184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530632633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"64f4f2ec-7112-49b8-a86f-8acd428f967e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530656713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530666322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530676182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530686022Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530695722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530705871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530715891Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:39415)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530774069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:41535)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530838048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:53338)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.530849557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531388652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531504989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531538208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531670534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531777601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[0]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 120, 120, 131, 131, 131, 133, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.53182429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[0]R generation[0..12]: [73, 77, 78, 68, 78, 72, 72, 78, 78, 78, 79, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.53183557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531846829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[1]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 120, 120, 131, 131, 131, 133, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531857879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[1]R generation[0..12]: [73, 77, 78, 68, 78, 72, 72, 78, 78, 78, 79, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531868189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531879158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[2]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 120, 120, 131, 131, 131, 133, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531889908Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[2]R generation[0..12]: [73, 77, 78, 68, 78, 72, 72, 78, 78, 78, 79, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531900178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531911157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531920437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531931817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531941077Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531950406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"":"downstairs","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531960146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531968816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"0","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531978536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"1","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531988095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"client":"2","":"downstairs","session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.531997965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532007115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"64f4f2ec-7112-49b8-a86f-8acd428f967e is now active with session: b74096f1-b786-470c-abdc-e938ac4e5f8d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532017154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532025874Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"session_id":"b74096f1-b786-470c-abdc-e938ac4e5f8d"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532034994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Activated sub_volume 64f4f2ec-7112-49b8-a86f-8acd428f967e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532042894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532051023Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.532396304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} 01/30 Write block 138 len 32768 data: 7 7 6 7 6 6 7 7 02/30 Read block 45 len 28672 03/30 Read block 209 len 24576 04/30 Write block 110 len 20480 data: 6 7 6 7 5 05/30 Read block 56 len 16384 06/30 Read block 572 len 28672 07/30 Read block 137 len 32768 08/30 Flush 09/30 Write block 284 len 20480 data: 3 3 4 4 3 10/30 Write block 116 len 24576 data: 6 8 10 12 9 11 11/30 Write block 371 len 8192 data: 7 6 12/30 Write block 445 len 20480 data: 11 9 7 6 7 13/30 Read block 161 len 32768 14/30 Write block 508 len 20480 data: 4 4 4 5 6 15/30 Read block 155 len 36864 16/30 Read block 52 len 12288 17/30 Read block 423 len 40960 18/30 Write block 218 len 36864 data: 5 5 7 9 7 7 7 9 9 19/30 Write block 397 len 12288 data: 8 7 5 20/30 Read block 236 len 32768 21/30 Read block 86 len 4096 22/30 Read block 354 len 12288 23/30 Read block 257 len 24576 24/30 Read block 402 len 40960 25/30 Read block 99 len 24576 26/30 Read block 61 len 4096 27/30 Write block 204 len 32768 data: 8 7 5 3 5 9 8 7 28/30 Read block 411 len 24576 29/30 Read block 175 len 40960 30/30 Write block 388 len 28672 data: 9 9 7 5 8 9 8 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 3 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 25 1 27 Done 29 5 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) None Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:45.541491418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:45.541672623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1456,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 78 78 78 131 131 131 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 78 132 132 132 F F F 5 100-119 80 72 80 134 120 134 T T T <--- 6 120-139 80 72 80 134 120 134 T T T <--- 7 140-159 80 78 80 134 131 134 F T F <--- 8 160-179 78 78 78 131 131 131 F F F 9 180-199 78 78 78 131 131 131 F F F 10 200-219 79 79 79 133 133 133 T F T <--- 11 220-239 78 78 78 132 132 132 T F T <--- 12 240-259 76 76 76 127 127 127 F F F 13 260-279 76 76 76 127 127 127 F F F 14 280-299 65 65 65 106 106 106 T F T <--- 15 300-319 78 78 78 131 131 131 F F F 16 320-339 74 74 74 125 125 125 F F F 17 340-359 78 78 78 129 129 129 F F F 18 360-379 78 78 78 129 129 129 T F T <--- 19 380-399 76 76 76 127 127 127 T F T <--- 20 400-419 74 74 74 123 123 123 F F F 21 420-439 78 78 78 129 129 129 F F F 22 440-459 78 78 78 131 131 131 T F T <--- 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F F F 25 500-519 76 76 76 127 127 127 T F T <--- 26 520-539 76 76 76 127 127 127 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 80, Max flush: 134 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 78 78 78 131 131 131 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 78 132 132 132 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 80 80 80 134 134 134 F F F 8 160-179 78 78 78 131 131 131 F F F 9 180-199 78 78 78 131 131 131 F F F 10 200-219 81 81 81 135 135 135 F F F 11 220-239 81 81 81 135 135 135 F F F 12 240-259 76 76 76 127 127 127 F F F 13 260-279 76 76 76 127 127 127 F F F 14 280-299 81 81 81 135 135 135 F F F 15 300-319 78 78 78 131 131 131 F F F 16 320-339 74 74 74 125 125 125 F F F 17 340-359 78 78 78 129 129 129 F F F 18 360-379 81 81 81 135 135 135 F F F 19 380-399 81 81 81 135 135 135 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 78 78 78 129 129 129 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 75 75 75 126 126 126 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 81, Max flush: 135 Begin loop 41 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.794134682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802236624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802529806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802555726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802564125Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802575785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Crucible 2a7452db-7713-4112-9591-88bb22850def has session id: 22cc3085-402e-48ae-9da9-9448daee279d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802907216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Upstairs opts: Upstairs UUID: 2a7452db-7713-4112-9591-88bb22850def, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802935195Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Crucible stats registered with UUID: 2a7452db-7713-4112-9591-88bb22850def","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.802945295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:49.802954504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803126039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803156229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803167238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803268765Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803286575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"2a7452db-7713-4112-9591-88bb22850def active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803305024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803316294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803326284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803336164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803345753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803355853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803365763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46359)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.803439821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"ds_connection connected from Ok(127.0.0.1:60519)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.80346532Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"ds_connection connected from Ok(127.0.0.1:59360)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.80347673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804153871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.80418693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804204049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804482011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804495631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[0]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 135, 135, 134, 131, 131, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.80452793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[0]R generation[0..12]: [73, 77, 78, 68, 78, 81, 81, 80, 78, 78, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.80454098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804552089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[1]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 135, 135, 134, 131, 131, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804562819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[1]R generation[0..12]: [73, 77, 78, 68, 78, 81, 81, 80, 78, 78, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804573349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804584418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[2]R flush_numbers[0..12]: [122, 128, 131, 112, 132, 135, 135, 134, 131, 131, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804595188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[2]R generation[0..12]: [73, 77, 78, 68, 78, 81, 81, 80, 78, 78, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804605628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804616848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804626477Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804636307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Next flush: 136","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804645727Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804655226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"":"downstairs","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804665096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804674156Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"0","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804683986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"1","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804694895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"client":"2","":"downstairs","session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804705015Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804714125Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"2a7452db-7713-4112-9591-88bb22850def is now active with session: 22cc3085-402e-48ae-9da9-9448daee279d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804724265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804733084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"session_id":"22cc3085-402e-48ae-9da9-9448daee279d"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804742264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Activated sub_volume 2a7452db-7713-4112-9591-88bb22850def","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804750364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.804758424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.805102324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} 01/30 Write block 269 len 8192 data: 9 11 02/30 Write block 422 len 8192 data: 6 8 03/30 Read block 421 len 16384 04/30 Read block 567 len 20480 05/30 Read block 172 len 8192 06/30 Write block 320 len 40960 data: 6 8 7 9 9 8 10 10 9 11 07/30 Write block 422 len 24576 data: 7 9 8 10 10 7 08/30 Write block 547 len 36864 data: 4 4 4 6 6 8 8 8 9 09/30 Flush 10/30 Write block 56 len 8192 data: 8 9 11/30 Flush 12/30 Write block 275 len 16384 data: 9 8 10 9 13/30 Read block 576 len 32768 14/30 Read block 219 len 40960 15/30 Read block 455 len 28672 16/30 Read block 59 len 28672 17/30 Write block 173 len 36864 data: 5 6 6 5 6 6 6 6 6 18/30 Read block 55 len 4096 19/30 Read block 521 len 4096 20/30 Write block 291 len 36864 data: 4 6 8 5 4 5 5 6 5 21/30 Write block 550 len 16384 data: 7 7 9 9 22/30 Flush 23/30 Write block 4 len 24576 data: 7 6 7 9 9 6 24/30 Write block 341 len 24576 data: 6 5 5 5 5 6 25/30 Read block 103 len 12288 26/30 Read block 279 len 16384 27/30 Read block 131 len 32768 28/30 Read block 47 len 32768 29/30 Write block 333 len 32768 data: 6 6 8 7 6 4 5 6 30/30 Write block 242 len 36864 data: 5 5 5 4 7 7 10 10 12 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Read 4 Done Sent Done false 1003 Acked Read 5 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:49.813642144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:49.813824729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1464,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 122 122 122 T F T <--- 1 020-039 77 77 77 128 128 128 F F F 2 040-059 82 78 82 137 131 137 F F F <--- 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 78 132 132 132 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 80 80 80 134 134 134 F F F 8 160-179 82 78 82 138 131 138 F F F <--- 9 180-199 82 78 82 138 131 138 F F F <--- 10 200-219 81 81 81 135 135 135 F F F 11 220-239 81 81 81 135 135 135 F F F 12 240-259 76 76 76 127 127 127 T F T <--- 13 260-279 82 76 82 138 127 138 F T F <--- 14 280-299 82 81 82 138 135 138 F F F <--- 15 300-319 78 78 78 131 131 131 F F F 16 320-339 82 74 82 136 125 136 T F T <--- 17 340-359 78 78 78 129 129 129 T F T <--- 18 360-379 81 81 81 135 135 135 F F F 19 380-399 81 81 81 135 135 135 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 82 78 82 136 129 136 F F F <--- 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 82 75 82 138 126 138 F F F <--- 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 82, Max flush: 138 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 82 82 82 137 137 137 F F F 3 060-079 68 68 68 112 112 112 F F F 4 080-099 78 78 78 132 132 132 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 80 80 80 134 134 134 F F F 8 160-179 82 82 82 138 138 138 F F F 9 180-199 82 82 82 138 138 138 F F F 10 200-219 81 81 81 135 135 135 F F F 11 220-239 81 81 81 135 135 135 F F F 12 240-259 83 83 83 139 139 139 F F F 13 260-279 82 82 82 138 138 138 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 78 78 78 131 131 131 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 83 83 83 139 139 139 F F F 18 360-379 81 81 81 135 135 135 F F F 19 380-399 81 81 81 135 135 135 F F F 20 400-419 74 74 74 123 123 123 F F F 21 420-439 82 82 82 136 136 136 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 76 76 76 127 127 127 F F F 27 540-559 82 82 82 138 138 138 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 83, Max flush: 139 Begin loop 42 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.06690424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.075601875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.075911037Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.075937116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.075945896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.075957945Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Crucible 5e716951-d0b7-4ef3-94bd-4b41e93747e6 has session id: cd3162cf-b4c6-4925-906d-66bf0bf0d7cf","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076348584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Upstairs opts: Upstairs UUID: 5e716951-d0b7-4ef3-94bd-4b41e93747e6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076373274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Crucible stats registered with UUID: 5e716951-d0b7-4ef3-94bd-4b41e93747e6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076383213Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:54.076392263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076557239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076573558Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076583728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076719414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076735214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"5e716951-d0b7-4ef3-94bd-4b41e93747e6 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076771033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076780552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076790392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076800142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076809831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076819811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076831261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:61619)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.076985647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:53447)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.077030185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:47201)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.077042185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.0775806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.077620069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.077703656Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.077890521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.07791297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[0]R flush_numbers[0..12]: [139, 128, 137, 112, 132, 135, 135, 134, 138, 138, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078031607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[0]R generation[0..12]: [83, 77, 82, 68, 78, 81, 81, 80, 82, 82, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078052917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078064526Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[1]R flush_numbers[0..12]: [139, 128, 137, 112, 132, 135, 135, 134, 138, 138, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078075386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[1]R generation[0..12]: [83, 77, 82, 68, 78, 81, 81, 80, 82, 82, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078085766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078096975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[2]R flush_numbers[0..12]: [139, 128, 137, 112, 132, 135, 135, 134, 138, 138, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078110525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[2]R generation[0..12]: [83, 77, 82, 68, 78, 81, 81, 80, 82, 82, 81, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078121035Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078131974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078141544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078151254Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078160604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078170193Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"":"downstairs","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078179953Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078188873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"0","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078198782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"1","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078208572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"client":"2","":"downstairs","session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078218492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078227662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"5e716951-d0b7-4ef3-94bd-4b41e93747e6 is now active with session: cd3162cf-b4c6-4925-906d-66bf0bf0d7cf","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078237801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078246691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"session_id":"cd3162cf-b4c6-4925-906d-66bf0bf0d7cf"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078257541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Activated sub_volume 5e716951-d0b7-4ef3-94bd-4b41e93747e6","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078265681Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.07827387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.078603021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} 01/30 Write block 533 len 36864 data: 6 6 6 8 10 8 6 7 7 02/30 Read block 433 len 8192 03/30 Write block 376 len 24576 data: 8 9 6 6 6 7 04/30 Write block 42 len 12288 data: 12 13 12 05/30 Read block 278 len 36864 06/30 Write block 207 len 40960 data: 4 6 10 9 8 8 6 5 5 6 07/30 Read block 425 len 16384 08/30 Write block 145 len 28672 data: 8 6 3 6 9 8 10 09/30 Read block 23 len 12288 10/30 Read block 225 len 20480 11/30 Flush 12/30 Write block 435 len 8192 data: 8 6 13/30 Write block 250 len 12288 data: 13 12 11 14/30 Write block 415 len 32768 data: 5 4 6 5 5 7 5 8 15/30 Write block 490 len 32768 data: 6 6 7 7 6 7 7 8 16/30 Read block 319 len 36864 17/30 Read block 573 len 40960 18/30 Read block 312 len 4096 19/30 Read block 515 len 32768 20/30 Read block 249 len 16384 21/30 Read block 113 len 24576 22/30 Write block 545 len 40960 data: 4 4 5 5 5 8 8 10 10 9 23/30 Read block 170 len 16384 24/30 Write block 278 len 8192 data: 10 8 25/30 Write block 96 len 4096 data: 6 26/30 Read block 67 len 20480 27/30 Write block 237 len 12288 data: 6 7 8 28/30 Write block 192 len 32768 data: 6 8 7 8 7 7 9 11 29/30 Read block 231 len 28672 30/30 Write block 54 len 40960 data: 8 7 9 10 8 9 6 5 4 5 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 1 26 Done 6 29 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:54.087917809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:54.088095194Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1472,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 82 84 84 137 140 140 T T T <--- 3 060-079 68 68 68 112 112 112 F T T <--- 4 080-099 78 78 78 132 132 132 F T T <--- 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 80 84 84 134 140 140 F F F <--- 8 160-179 82 82 82 138 138 138 F F F 9 180-199 82 82 82 138 138 138 F T T <--- 10 200-219 81 84 84 135 140 140 T F F <--- 11 220-239 81 81 81 135 135 135 F T T <--- 12 240-259 83 83 83 139 139 139 F T T <--- 13 260-279 82 82 82 138 138 138 F T T <--- 14 280-299 82 82 82 138 138 138 F F F 15 300-319 78 78 78 131 131 131 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 83 83 83 139 139 139 F F F 18 360-379 81 84 84 135 140 140 T F F <--- 19 380-399 81 84 84 135 140 140 T F F <--- 20 400-419 74 74 74 123 123 123 F T T <--- 21 420-439 82 82 82 136 136 136 F T T <--- 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 78 78 78 131 131 131 F T T <--- 25 500-519 81 81 81 135 135 135 F F F 26 520-539 76 84 84 127 140 140 T F F <--- 27 540-559 82 84 84 138 140 140 T T T <--- 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 84, Max flush: 140 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 85 85 85 141 141 141 F F F 3 060-079 85 85 85 141 141 141 F F F 4 080-099 85 85 85 141 141 141 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 84 84 84 140 140 140 F F F 8 160-179 82 82 82 138 138 138 F F F 9 180-199 85 85 85 141 141 141 F F F 10 200-219 84 84 84 140 140 140 F F F 11 220-239 85 85 85 141 141 141 F F F 12 240-259 85 85 85 141 141 141 F F F 13 260-279 85 85 85 141 141 141 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 78 78 78 131 131 131 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 83 83 83 139 139 139 F F F 18 360-379 84 84 84 140 140 140 F F F 19 380-399 84 84 84 140 140 140 F F F 20 400-419 85 85 85 141 141 141 F F F 21 420-439 85 85 85 141 141 141 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 85 85 85 141 141 141 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 84 84 84 140 140 140 F F F 27 540-559 85 85 85 141 141 141 F F F 28 560-579 76 76 76 127 127 127 F F F 29 580-599 69 69 69 114 114 114 F F F Max gen: 85, Max flush: 141 Begin loop 43 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.362150836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:58.369328365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.369354064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376158613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376428475Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376463854Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376473714Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376485883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Crucible 672b2b36-0fb8-4976-a944-7e5c53929578 has session id: 28c46c38-c560-42c2-ac23-76ea85839c3f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376797375Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Upstairs opts: Upstairs UUID: 672b2b36-0fb8-4976-a944-7e5c53929578, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376855273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Crucible stats registered with UUID: 672b2b36-0fb8-4976-a944-7e5c53929578","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.376867943Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:58.376877222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377028748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377045218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377055297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377180314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377193894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"672b2b36-0fb8-4976-a944-7e5c53929578 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377212453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377221183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377230913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377240562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377250122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377260092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377270121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:42663)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.3773309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58205)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377357129Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62243)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.377371069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.37803969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378058989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378086818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.37838005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.37839152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[0]R flush_numbers[0..12]: [139, 128, 141, 141, 141, 135, 135, 140, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378417039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[0]R generation[0..12]: [83, 77, 85, 85, 85, 81, 81, 84, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378428329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378439439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[1]R flush_numbers[0..12]: [139, 128, 141, 141, 141, 135, 135, 140, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378450168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[1]R generation[0..12]: [83, 77, 85, 85, 85, 81, 81, 84, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378460648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378471738Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[2]R flush_numbers[0..12]: [139, 128, 141, 141, 141, 135, 135, 140, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378482687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[2]R generation[0..12]: [83, 77, 85, 85, 85, 81, 81, 84, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378493237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378506747Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378516516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378526196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Next flush: 142","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378535666Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378545306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"":"downstairs","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378555065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378563955Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"0","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378573755Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"1","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378583435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"client":"2","":"downstairs","session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378593274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378602474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"672b2b36-0fb8-4976-a944-7e5c53929578 is now active with session: 28c46c38-c560-42c2-ac23-76ea85839c3f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378612464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378621073Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"session_id":"28c46c38-c560-42c2-ac23-76ea85839c3f"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378630113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Activated sub_volume 672b2b36-0fb8-4976-a944-7e5c53929578","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378638313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378646513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.378959974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} 01/30 Write block 92 len 12288 data: 4 4 5 02/30 Write block 365 len 28672 data: 8 6 6 7 8 6 8 03/30 Read block 44 len 16384 04/30 Write block 397 len 24576 data: 9 8 6 7 7 7 05/30 Read block 59 len 20480 06/30 Write block 255 len 20480 data: 8 7 5 5 6 07/30 Write block 417 len 32768 data: 7 6 6 8 6 9 10 9 08/30 Read block 103 len 36864 09/30 Read block 341 len 16384 10/30 Write block 303 len 12288 data: 5 6 5 11/30 Write block 579 len 4096 data: 8 12/30 Read block 268 len 12288 13/30 Flush 14/30 Write block 44 len 20480 data: 13 11 13 16 11 15/30 Write block 581 len 24576 data: 7 10 9 9 9 8 16/30 Write block 351 len 24576 data: 8 10 9 7 6 5 17/30 Read block 231 len 12288 18/30 Read block 365 len 4096 19/30 Write block 341 len 4096 data: 7 20/30 Write block 365 len 16384 data: 9 7 7 8 21/30 Write block 560 len 16384 data: 7 8 9 7 22/30 Read block 318 len 8192 23/30 Read block 505 len 4096 24/30 Read block 573 len 8192 25/30 Read block 409 len 40960 26/30 Read block 596 len 8192 27/30 Read block 408 len 40960 28/30 Write block 144 len 32768 data: 8 9 7 4 7 10 9 11 29/30 Read block 70 len 28672 30/30 Read block 395 len 4096 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Write 7 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 1 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:11:58.386990508Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:11:58.387154634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1480,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 85 85 85 141 141 141 F T T <--- 3 060-079 85 85 85 141 141 141 F F F 4 080-099 85 86 86 141 142 142 F F F <--- 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 84 84 84 140 140 140 F T T <--- 8 160-179 82 82 82 138 138 138 F F F 9 180-199 85 85 85 141 141 141 F F F 10 200-219 84 84 84 140 140 140 F F F 11 220-239 85 85 85 141 141 141 F F F 12 240-259 85 86 86 141 142 142 F F F <--- 13 260-279 85 85 85 141 141 141 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 78 86 86 131 142 142 F F F <--- 16 320-339 83 83 83 139 139 139 F F F 17 340-359 83 83 83 139 139 139 F T T <--- 18 360-379 84 86 86 140 142 142 F T T <--- 19 380-399 84 86 86 140 142 142 F F F <--- 20 400-419 85 86 86 141 142 142 F F F <--- 21 420-439 85 86 86 141 142 142 F F F <--- 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 85 85 85 141 141 141 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 84 84 84 140 140 140 F F F 27 540-559 85 85 85 141 141 141 F F F 28 560-579 76 86 86 127 142 142 F T T <--- 29 580-599 69 69 69 114 114 114 F T T <--- Max gen: 86, Max flush: 142 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 F F F 1 020-039 77 77 77 128 128 128 F F F 2 040-059 87 87 87 143 143 143 F F F 3 060-079 85 85 85 141 141 141 F F F 4 080-099 86 86 86 142 142 142 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 87 87 87 143 143 143 F F F 8 160-179 82 82 82 138 138 138 F F F 9 180-199 85 85 85 141 141 141 F F F 10 200-219 84 84 84 140 140 140 F F F 11 220-239 85 85 85 141 141 141 F F F 12 240-259 86 86 86 142 142 142 F F F 13 260-279 85 85 85 141 141 141 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 86 86 86 142 142 142 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 87 87 87 143 143 143 F F F 19 380-399 86 86 86 142 142 142 F F F 20 400-419 86 86 86 142 142 142 F F F 21 420-439 86 86 86 142 142 142 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 F F F 24 480-499 85 85 85 141 141 141 F F F 25 500-519 81 81 81 135 135 135 F F F 26 520-539 84 84 84 140 140 140 F F F 27 540-559 85 85 85 141 141 141 F F F 28 560-579 87 87 87 143 143 143 F F F 29 580-599 87 87 87 143 143 143 F F F Max gen: 87, Max flush: 143 Begin loop 44 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.643833384Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652056753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652360435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652386434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652394744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652406453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Crucible 6d430c38-0043-4415-b6c8-075b4764a47f has session id: ea3fa491-fbb9-499c-b9df-72bd833f44dd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652757153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Upstairs opts: Upstairs UUID: 6d430c38-0043-4415-b6c8-075b4764a47f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652787223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Crucible stats registered with UUID: 6d430c38-0043-4415-b6c8-075b4764a47f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652798412Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:02.652807732Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652961818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652978647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.652988937Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653120493Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653134623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"6d430c38-0043-4415-b6c8-075b4764a47f active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653168742Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653178122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653188091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653197831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653207471Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65321758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65322784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:36381)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653267499Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:63875)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653278679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:40247)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.653412275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654011388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654037757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654102646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654377868Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654391897Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[0]R flush_numbers[0..12]: [139, 128, 143, 141, 142, 135, 135, 143, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654415837Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[0]R generation[0..12]: [83, 77, 87, 85, 86, 81, 81, 87, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654441996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654454136Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[1]R flush_numbers[0..12]: [139, 128, 143, 141, 142, 135, 135, 143, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654465025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[1]R generation[0..12]: [83, 77, 87, 85, 86, 81, 81, 87, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654475615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654486555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[2]R flush_numbers[0..12]: [139, 128, 143, 141, 142, 135, 135, 143, 138, 141, 140, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654497264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[2]R generation[0..12]: [83, 77, 87, 85, 86, 81, 81, 87, 82, 85, 84, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654507664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654518794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654528154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654538043Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Next flush: 144","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654547573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654557023Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"":"downstairs","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654569182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654578082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"0","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654593152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"1","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654602942Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"client":"2","":"downstairs","session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654613001Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654622171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"6d430c38-0043-4415-b6c8-075b4764a47f is now active with session: ea3fa491-fbb9-499c-b9df-72bd833f44dd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.654632261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65464108Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"session_id":"ea3fa491-fbb9-499c-b9df-72bd833f44dd"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65465043Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Activated sub_volume 6d430c38-0043-4415-b6c8-075b4764a47f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65465848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65466658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.65501553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} 01/30 Write block 267 len 32768 data: 7 7 10 12 8 8 8 8 02/30 Read block 251 len 8192 03/30 Flush 04/30 Write block 540 len 4096 data: 8 05/30 Read block 314 len 20480 06/30 Read block 23 len 28672 07/30 Read block 46 len 20480 08/30 Read block 554 len 12288 09/30 Write block 145 len 32768 data: 10 8 5 8 11 10 12 11 10/30 Read block 97 len 20480 11/30 Read block 143 len 4096 12/30 Write block 192 len 40960 data: 7 9 8 9 8 8 10 12 9 9 13/30 Read block 338 len 20480 14/30 Write block 478 len 20480 data: 3 3 3 3 6 15/30 Write block 547 len 28672 data: 6 6 6 9 9 11 11 16/30 Write block 550 len 16384 data: 10 10 12 12 17/30 Read block 239 len 24576 18/30 Read block 582 len 28672 19/30 Read block 95 len 20480 20/30 Read block 48 len 16384 21/30 Write block 78 len 40960 data: 6 7 5 6 6 6 6 5 5 7 22/30 Write block 94 len 16384 data: 6 6 7 5 23/30 Read block 474 len 36864 24/30 Read block 480 len 36864 25/30 Write block 511 len 36864 data: 6 7 7 8 7 8 8 10 11 26/30 Write block 402 len 12288 data: 8 6 7 27/30 Write block 525 len 40960 data: 7 7 8 6 6 7 8 8 7 7 28/30 Write block 14 len 36864 data: 9 8 8 6 8 10 9 9 10 29/30 Read block 532 len 4096 30/30 Read block 448 len 24576 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Read 5 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1002)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:02.664081795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:02.66425165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1488,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 139 139 139 T T F <--- 1 020-039 77 77 77 128 128 128 T T F <--- 2 040-059 87 87 87 143 143 143 F F F 3 060-079 85 85 85 141 141 141 T T F <--- 4 080-099 86 86 86 142 142 142 T T F <--- 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 87 87 87 143 143 143 T T F <--- 8 160-179 82 82 82 138 138 138 F F F 9 180-199 85 85 85 141 141 141 T T F <--- 10 200-219 84 84 84 140 140 140 T T F <--- 11 220-239 85 85 85 141 141 141 F F F 12 240-259 86 86 86 142 142 142 F F F 13 260-279 88 88 85 144 144 141 F F T <--- 14 280-299 82 82 82 138 138 138 F F F 15 300-319 86 86 86 142 142 142 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 87 87 87 143 143 143 F F F 19 380-399 86 86 86 142 142 142 F F F 20 400-419 86 86 86 142 142 142 T T F <--- 21 420-439 86 86 86 142 142 142 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 76 76 76 127 127 127 T T F <--- 24 480-499 85 85 85 141 141 141 T T F <--- 25 500-519 81 81 81 135 135 135 T T F <--- 26 520-539 84 84 84 140 140 140 T T F <--- 27 540-559 85 85 85 141 141 141 T T F <--- 28 560-579 87 87 87 143 143 143 F F F 29 580-599 87 87 87 143 143 143 F F F Max gen: 88, Max flush: 144 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 145 145 145 F F F 1 020-039 89 89 89 145 145 145 F F F 2 040-059 87 87 87 143 143 143 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 81 81 81 135 135 135 F F F 7 140-159 89 89 89 145 145 145 F F F 8 160-179 82 82 82 138 138 138 F F F 9 180-199 89 89 89 145 145 145 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 85 85 85 141 141 141 F F F 12 240-259 86 86 86 142 142 142 F F F 13 260-279 88 88 88 144 144 144 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 86 86 86 142 142 142 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 87 87 87 143 143 143 F F F 19 380-399 86 86 86 142 142 142 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 86 86 86 142 142 142 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 89 89 89 145 145 145 F F F 25 500-519 89 89 89 145 145 145 F F F 26 520-539 89 89 89 145 145 145 F F F 27 540-559 89 89 89 145 145 145 F F F 28 560-579 87 87 87 143 143 143 F F F 29 580-599 87 87 87 143 143 143 F F F Max gen: 89, Max flush: 145 Begin loop 45 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.932886294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941081474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941380185Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941415324Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941427354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941439514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Crucible d5712990-6e57-4fff-a43c-392e6151bd26 has session id: a9a6fc41-f722-48a1-aa8d-efeef1d71094","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941753445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Upstairs opts: Upstairs UUID: d5712990-6e57-4fff-a43c-392e6151bd26, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941778564Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Crucible stats registered with UUID: d5712990-6e57-4fff-a43c-392e6151bd26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941788664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:06.941797874Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941959239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.941989728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942000818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942128694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942142624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"d5712990-6e57-4fff-a43c-392e6151bd26 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942177713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942187153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942196922Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942206622Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942218302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942228362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942238291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"ds_connection connected from Ok(127.0.0.1:54388)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94230311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"ds_connection connected from Ok(127.0.0.1:49221)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942329139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"ds_connection connected from Ok(127.0.0.1:40759)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942387247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.942957411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94298529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943122346Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943269092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943287282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[0]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 135, 145, 138, 145, 145, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943476927Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[0]R generation[0..12]: [89, 89, 87, 89, 89, 81, 81, 89, 82, 89, 89, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943488416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943499636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[1]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 135, 145, 138, 145, 145, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943510506Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[1]R generation[0..12]: [89, 89, 87, 89, 89, 81, 81, 89, 82, 89, 89, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943523455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943534485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[2]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 135, 145, 138, 145, 145, 141]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943545235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[2]R generation[0..12]: [89, 89, 87, 89, 89, 81, 81, 89, 82, 89, 89, 85]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943555664Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943566634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943576334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943586133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Next flush: 146","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943595603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943605033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"":"downstairs","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943614913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943623792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"0","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943633712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"1","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943643472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"client":"2","":"downstairs","session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943653582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943662871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"d5712990-6e57-4fff-a43c-392e6151bd26 is now active with session: a9a6fc41-f722-48a1-aa8d-efeef1d71094","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943675271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.943684531Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94369248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"session_id":"a9a6fc41-f722-48a1-aa8d-efeef1d71094"} {"msg":"Activated sub_volume d5712990-6e57-4fff-a43c-392e6151bd26","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94370224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94371058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.94406029Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} 01/30 Write block 578 len 28672 data: 10 9 7 8 11 10 10 02/30 Write block 268 len 32768 data: 8 11 13 9 9 9 9 10 03/30 Read block 345 len 16384 04/30 Write block 576 len 16384 data: 7 9 11 10 05/30 Write block 528 len 20480 data: 7 7 8 9 9 06/30 Read block 5 len 4096 07/30 Write block 504 len 4096 data: 7 08/30 Write block 375 len 4096 data: 8 09/30 Read block 279 len 24576 10/30 Read block 50 len 28672 11/30 Read block 0 len 16384 12/30 Read block 394 len 16384 13/30 Write block 568 len 32768 data: 8 6 7 4 3 4 4 6 14/30 Write block 228 len 4096 data: 7 15/30 Read block 19 len 12288 16/30 Write block 123 len 20480 data: 10 8 9 9 9 17/30 Read block 380 len 28672 18/30 Read block 309 len 32768 19/30 Write block 590 len 40960 data: 6 6 6 4 5 3 3 3 3 2 20/30 Read block 132 len 8192 21/30 Read block 198 len 24576 22/30 Flush 23/30 Write block 194 len 20480 data: 9 10 9 9 11 24/30 Read block 229 len 16384 25/30 Flush 26/30 Write block 383 len 12288 data: 8 9 9 27/30 Write block 160 len 20480 data: 6 8 9 9 9 28/30 Read block 307 len 36864 29/30 Read block 472 len 20480 30/30 Write block 307 len 32768 data: 7 7 6 6 6 6 6 5 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Write 5 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 29 29 2 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:06.952686978Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:06.952872032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1496,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 145 145 145 F F F 1 020-039 89 89 89 145 145 145 F F F 2 040-059 87 87 87 143 143 143 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 81 146 146 135 F F F <--- 7 140-159 89 89 89 145 145 145 F F F 8 160-179 82 82 82 138 138 138 T T F <--- 9 180-199 90 90 89 147 147 145 F F F <--- 10 200-219 89 89 89 145 145 145 F F F 11 220-239 90 90 85 146 146 141 F F F <--- 12 240-259 86 86 86 142 142 142 F F F 13 260-279 90 90 88 146 146 144 F F T <--- 14 280-299 82 82 82 138 138 138 F F F 15 300-319 86 86 86 142 142 142 T T F <--- 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 90 90 87 146 146 143 F F F <--- 19 380-399 86 86 86 142 142 142 T T F <--- 20 400-419 89 89 89 145 145 145 F F F 21 420-439 86 86 86 142 142 142 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 89 89 89 145 145 145 F F F 25 500-519 90 90 89 146 146 145 F F F <--- 26 520-539 90 90 89 146 146 145 F F F <--- 27 540-559 89 89 89 145 145 145 F F F 28 560-579 90 90 87 146 146 143 F F T <--- 29 580-599 90 90 87 146 146 143 F F T <--- Max gen: 90, Max flush: 147 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 145 145 145 F F F 1 020-039 89 89 89 145 145 145 F F F 2 040-059 87 87 87 143 143 143 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 90 146 146 146 F F F 7 140-159 89 89 89 145 145 145 F F F 8 160-179 91 91 91 148 148 148 F F F 9 180-199 90 90 90 147 147 147 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 90 90 90 146 146 146 F F F 12 240-259 86 86 86 142 142 142 F F F 13 260-279 90 90 90 146 146 146 F F F 14 280-299 82 82 82 138 138 138 F F F 15 300-319 91 91 91 148 148 148 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 90 90 90 146 146 146 F F F 19 380-399 91 91 91 148 148 148 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 86 86 86 142 142 142 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 89 89 89 145 145 145 F F F 25 500-519 90 90 90 146 146 146 F F F 26 520-539 90 90 90 146 146 146 F F F 27 540-559 89 89 89 145 145 145 F F F 28 560-579 90 90 90 146 146 146 F F F 29 580-599 90 90 90 146 146 146 F F F Max gen: 91, Max flush: 148 Begin loop 46 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.207608948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.215761568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.21604888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.2160729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216081999Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216093819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Crucible a59b86c7-b227-47b4-a4f1-012bcf6bcbc7 has session id: ec214687-910b-4d10-8cea-7fda14b00ffd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.21643434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Upstairs opts: Upstairs UUID: a59b86c7-b227-47b4-a4f1-012bcf6bcbc7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216463909Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Crucible stats registered with UUID: a59b86c7-b227-47b4-a4f1-012bcf6bcbc7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216478278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:11.216487678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216659773Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216689802Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216700682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216815609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216829388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"a59b86c7-b227-47b4-a4f1-012bcf6bcbc7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216900276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216909996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216919786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216929576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216939005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216966395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216986714Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"ds_connection connected from Ok(127.0.0.1:38517)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.216997394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"ds_connection connected from Ok(127.0.0.1:55802)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217019373Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"ds_connection connected from Ok(127.0.0.1:58894)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217078701Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217676345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217727153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217869349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.217986936Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218026655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[0]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 146, 145, 148, 147, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218140472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[0]R generation[0..12]: [89, 89, 87, 89, 89, 81, 90, 89, 91, 90, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218151711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218162811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[1]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 146, 145, 148, 147, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218173551Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[1]R generation[0..12]: [89, 89, 87, 89, 89, 81, 90, 89, 91, 90, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.21818393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.21819521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[2]R flush_numbers[0..12]: [145, 145, 143, 145, 145, 135, 146, 145, 148, 147, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.21820601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[2]R generation[0..12]: [89, 89, 87, 89, 89, 81, 90, 89, 91, 90, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218216219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218229349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218238919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218248619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Next flush: 149","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218257918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218267188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"":"downstairs","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218282338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218291067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"0","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218300707Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"1","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218310417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"client":"2","":"downstairs","session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218320067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218329196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"a59b86c7-b227-47b4-a4f1-012bcf6bcbc7 is now active with session: ec214687-910b-4d10-8cea-7fda14b00ffd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218339196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218348076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"session_id":"ec214687-910b-4d10-8cea-7fda14b00ffd"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218357066Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Activated sub_volume a59b86c7-b227-47b4-a4f1-012bcf6bcbc7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218365025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218373025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.218700966Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} 01/30 Write block 38 len 16384 data: 8 9 11 10 02/30 Flush 03/30 Write block 431 len 32768 data: 10 6 6 7 9 7 6 6 04/30 Read block 281 len 36864 05/30 Write block 277 len 24576 data: 11 11 9 8 7 7 06/30 Write block 371 len 32768 data: 9 7 6 7 9 9 10 7 07/30 Read block 202 len 24576 08/30 Read block 156 len 36864 09/30 Write block 36 len 28672 data: 7 7 9 10 12 11 13 10/30 Write block 522 len 24576 data: 7 7 7 8 8 9 11/30 Write block 545 len 20480 data: 5 5 7 7 7 12/30 Read block 409 len 8192 13/30 Read block 281 len 40960 14/30 Read block 266 len 16384 15/30 Read block 493 len 4096 16/30 Read block 494 len 16384 17/30 Flush 18/30 Write block 51 len 12288 data: 10 8 9 19/30 Read block 462 len 40960 20/30 Read block 581 len 16384 21/30 Read block 19 len 20480 22/30 Read block 368 len 16384 23/30 Read block 499 len 24576 24/30 Read block 401 len 28672 25/30 Read block 44 len 20480 26/30 Write block 234 len 36864 data: 8 8 6 7 8 9 8 7 6 27/30 Read block 358 len 28672 28/30 Read block 187 len 4096 29/30 Read block 291 len 24576 30/30 Write block 364 len 20480 data: 6 10 8 8 9 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 8 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 6 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 6 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 24 26 Done 27 27 4 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:11.22815513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:11.228350275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1504,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 145 145 145 F F F 1 020-039 92 92 92 150 150 149 F F F <--- 2 040-059 92 92 92 150 150 149 T T F <--- 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 90 146 146 146 F F F 7 140-159 89 89 89 145 145 145 F F F 8 160-179 91 91 91 148 148 148 F F F 9 180-199 90 90 90 147 147 147 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 90 90 90 146 146 146 T T F <--- 12 240-259 86 86 86 142 142 142 T T F <--- 13 260-279 92 92 90 150 150 146 F F T <--- 14 280-299 92 92 82 150 150 138 F F T <--- 15 300-319 91 91 91 148 148 148 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 92 92 90 150 150 146 T T T <--- 19 380-399 91 91 91 148 148 148 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 92 92 86 150 150 142 F F T <--- 22 440-459 81 81 81 135 135 135 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 89 89 89 145 145 145 F F F 25 500-519 90 90 90 146 146 146 F F F 26 520-539 92 92 90 150 150 146 F F F <--- 27 540-559 92 92 89 150 150 145 F F F <--- 28 560-579 90 90 90 146 146 146 F F F 29 580-599 90 90 90 146 146 146 F F F Max gen: 92, Max flush: 150 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 145 145 145 F F F 1 020-039 92 92 92 150 150 150 F F F 2 040-059 93 93 93 151 151 151 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 90 146 146 146 F F F 7 140-159 89 89 89 145 145 145 F F F 8 160-179 91 91 91 148 148 148 F F F 9 180-199 90 90 90 147 147 147 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 93 93 93 151 151 151 F F F 12 240-259 93 93 93 151 151 151 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 92 92 92 150 150 150 F F F 15 300-319 91 91 91 148 148 148 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 93 93 93 151 151 151 F F F 19 380-399 91 91 91 148 148 148 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 81 81 81 135 135 135 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 89 89 89 145 145 145 F F F 25 500-519 90 90 90 146 146 146 F F F 26 520-539 92 92 92 150 150 150 F F F 27 540-559 92 92 92 150 150 150 F F F 28 560-579 90 90 90 146 146 146 F F F 29 580-599 90 90 90 146 146 146 F F F Max gen: 93, Max flush: 151 Begin loop 47 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.480851673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489008923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489307695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489332134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489340964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489353014Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Crucible 8a22da17-206a-428c-a3eb-bb6cc605d022 has session id: 0cdbac9b-2436-423f-b33e-c6704ff88181","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489708244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Upstairs opts: Upstairs UUID: 8a22da17-206a-428c-a3eb-bb6cc605d022, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489737273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Crucible stats registered with UUID: 8a22da17-206a-428c-a3eb-bb6cc605d022","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489748623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:15.489758062Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489914038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489944967Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.489955927Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490131202Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490144122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"8a22da17-206a-428c-a3eb-bb6cc605d022 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490171031Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49018065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.4901909Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49020101Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49021063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490220619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490230719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"ds_connection connected from Ok(127.0.0.1:48786)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490334166Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"ds_connection connected from Ok(127.0.0.1:63633)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490359655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"ds_connection connected from Ok(127.0.0.1:48451)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490421694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.490988178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491005587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491026847Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491305689Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491330568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[0]R flush_numbers[0..12]: [145, 150, 151, 145, 145, 135, 146, 145, 148, 147, 145, 151]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491433585Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[0]R generation[0..12]: [89, 92, 93, 89, 89, 81, 90, 89, 91, 90, 89, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491444795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491456005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[1]R flush_numbers[0..12]: [145, 150, 151, 145, 145, 135, 146, 145, 148, 147, 145, 151]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491466674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[1]R generation[0..12]: [89, 92, 93, 89, 89, 81, 90, 89, 91, 90, 89, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491476994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491488084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[2]R flush_numbers[0..12]: [145, 150, 151, 145, 145, 135, 146, 145, 148, 147, 145, 151]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491498813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[2]R generation[0..12]: [89, 92, 93, 89, 89, 81, 90, 89, 91, 90, 89, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491509093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491519983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491529513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491539332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Next flush: 152","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491548712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491558032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"":"downstairs","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491568121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491577221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"0","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491588741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"1","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491598811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"client":"2","":"downstairs","session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49160882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49161814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"8a22da17-206a-428c-a3eb-bb6cc605d022 is now active with session: 0cdbac9b-2436-423f-b33e-c6704ff88181","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49162816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.49163674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"session_id":"0cdbac9b-2436-423f-b33e-c6704ff88181"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491645839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Activated sub_volume 8a22da17-206a-428c-a3eb-bb6cc605d022","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491653789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.491661919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.492060008Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} 01/30 Write block 494 len 32768 data: 7 8 8 9 8 5 5 6 02/30 Read block 385 len 16384 03/30 Read block 148 len 8192 04/30 Write block 588 len 8192 data: 9 7 05/30 Read block 113 len 40960 06/30 Read block 355 len 24576 07/30 Read block 331 len 32768 08/30 Read block 432 len 40960 09/30 Write block 176 len 36864 data: 6 7 7 7 7 7 7 8 9 10/30 Read block 460 len 36864 11/30 Read block 136 len 40960 12/30 Write block 155 len 8192 data: 7 7 13/30 Write block 37 len 20480 data: 8 10 11 13 12 14/30 Write block 15 len 4096 data: 9 15/30 Flush 16/30 Write block 445 len 28672 data: 12 10 8 7 8 6 7 17/30 Read block 93 len 32768 18/30 Read block 532 len 36864 19/30 Read block 107 len 36864 20/30 Read block 23 len 20480 21/30 Read block 369 len 24576 22/30 Write block 447 len 12288 data: 9 8 9 23/30 Read block 297 len 8192 24/30 Flush 25/30 Write block 309 len 36864 data: 7 7 7 7 7 6 6 6 7 26/30 Read block 593 len 12288 27/30 Write block 256 len 16384 data: 8 6 6 7 28/30 Read block 16 len 28672 29/30 Write block 235 len 24576 data: 9 7 8 9 10 9 30/30 Read block 358 len 8192 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 4 Done Sent Done false 1002 Acked Read 2 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 6 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:15.501430304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:15.501606429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1512,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 89 94 152 145 152 F F F <--- 1 020-039 94 92 94 152 150 152 F F F <--- 2 040-059 94 93 94 152 151 152 F F F <--- 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 90 146 146 146 F F F 7 140-159 94 89 94 152 145 152 F F F <--- 8 160-179 94 91 94 152 148 152 F F F <--- 9 180-199 94 90 94 152 147 152 F F F <--- 10 200-219 89 89 89 145 145 145 F F F 11 220-239 93 93 93 151 151 151 T F T <--- 12 240-259 93 93 93 151 151 151 T F T <--- 13 260-279 92 92 92 150 150 150 F F F 14 280-299 92 92 92 150 150 150 F F F 15 300-319 91 91 91 148 148 148 T F T <--- 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 93 93 93 151 151 151 F F F 19 380-399 91 91 91 148 148 148 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 81 94 153 135 153 F F F <--- 23 460-479 89 89 89 145 145 145 F F F 24 480-499 94 89 94 152 145 152 F T F <--- 25 500-519 94 90 94 152 146 152 F T F <--- 26 520-539 92 92 92 150 150 150 F F F 27 540-559 92 92 92 150 150 150 F F F 28 560-579 90 90 90 146 146 146 F F F 29 580-599 94 90 94 152 146 152 F F F <--- Max gen: 94, Max flush: 153 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 94 94 94 152 152 152 F F F 2 040-059 94 94 94 152 152 152 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 81 81 81 135 135 135 F F F 6 120-139 90 90 90 146 146 146 F F F 7 140-159 94 94 94 152 152 152 F F F 8 160-179 94 94 94 152 152 152 F F F 9 180-199 94 94 94 152 152 152 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 95 95 95 154 154 154 F F F 12 240-259 95 95 95 154 154 154 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 92 92 92 150 150 150 F F F 15 300-319 95 95 95 154 154 154 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 F F F 18 360-379 93 93 93 151 151 151 F F F 19 380-399 91 91 91 148 148 148 F F F 20 400-419 89 89 89 145 145 145 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 94 94 94 152 152 152 F F F 25 500-519 94 94 94 152 152 152 F F F 26 520-539 92 92 92 150 150 150 F F F 27 540-559 92 92 92 150 150 150 F F F 28 560-579 90 90 90 146 146 146 F F F 29 580-599 94 94 94 152 152 152 F F F Max gen: 95, Max flush: 154 Begin loop 48 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.758213852Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.766556717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.766890398Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.766925807Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.766935157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.766946786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Crucible 42954d23-20b3-437c-b5a2-9aa14b7fac8a has session id: 1ba2d4e2-20d2-4b18-9a5d-d323b9844284","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767318486Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Upstairs opts: Upstairs UUID: 42954d23-20b3-437c-b5a2-9aa14b7fac8a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767358395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Crucible stats registered with UUID: 42954d23-20b3-437c-b5a2-9aa14b7fac8a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767368665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:19.767377594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76752325Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76754084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76755083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767686266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767705295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"42954d23-20b3-437c-b5a2-9aa14b7fac8a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767731794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767741074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767751004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767760704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767770253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767780253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767792003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"ds_connection connected from Ok(127.0.0.1:32791)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.767855401Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"ds_connection connected from Ok(127.0.0.1:56690)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76788091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"ds_connection connected from Ok(127.0.0.1:57171)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76789244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768506693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768532262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768621399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768835933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768845713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[0]R flush_numbers[0..12]: [152, 152, 152, 145, 145, 135, 146, 152, 152, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768920591Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[0]R generation[0..12]: [94, 94, 94, 89, 89, 81, 90, 94, 94, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768931671Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76894289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[1]R flush_numbers[0..12]: [152, 152, 152, 145, 145, 135, 146, 152, 152, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76895395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[1]R generation[0..12]: [94, 94, 94, 89, 89, 81, 90, 94, 94, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.76896441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768978229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[2]R flush_numbers[0..12]: [152, 152, 152, 145, 145, 135, 146, 152, 152, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768989079Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[2]R generation[0..12]: [94, 94, 94, 89, 89, 81, 90, 94, 94, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.768999469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769010428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769020048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769029948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Next flush: 155","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769039358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769048787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"":"downstairs","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769058677Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769067467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"0","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769077027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"1","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769086736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"client":"2","":"downstairs","session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769096596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769105686Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"42954d23-20b3-437c-b5a2-9aa14b7fac8a is now active with session: 1ba2d4e2-20d2-4b18-9a5d-d323b9844284","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769115736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769125755Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"session_id":"1ba2d4e2-20d2-4b18-9a5d-d323b9844284"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769134855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Activated sub_volume 42954d23-20b3-437c-b5a2-9aa14b7fac8a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769142775Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769150905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.769507445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} 01/30 Write block 311 len 16384 data: 8 8 8 7 02/30 Write block 30 len 40960 data: 8 8 7 7 10 9 8 9 11 12 03/30 Flush 04/30 Write block 383 len 16384 data: 9 10 10 8 05/30 Write block 159 len 8192 data: 4 7 06/30 Read block 483 len 36864 07/30 Write block 559 len 12288 data: 7 8 9 08/30 Read block 116 len 4096 09/30 Read block 1 len 8192 10/30 Read block 382 len 32768 11/30 Read block 12 len 16384 12/30 Read block 457 len 16384 13/30 Write block 240 len 40960 data: 10 8 7 6 6 5 8 8 11 11 14/30 Write block 94 len 8192 data: 7 7 15/30 Read block 102 len 8192 16/30 Write block 116 len 40960 data: 7 9 11 13 10 12 10 11 9 10 17/30 Read block 217 len 32768 18/30 Read block 342 len 24576 19/30 Write block 520 len 20480 data: 11 9 8 8 8 20/30 Write block 408 len 36864 data: 7 7 6 7 6 5 5 6 5 21/30 Read block 384 len 32768 22/30 Write block 113 len 40960 data: 8 6 5 8 10 12 14 11 13 11 23/30 Write block 556 len 20480 data: 10 9 8 8 9 24/30 Read block 491 len 4096 25/30 Write block 249 len 12288 data: 12 14 13 26/30 Write block 358 len 16384 data: 6 8 6 6 27/30 Write block 22 len 4096 data: 11 28/30 Write block 165 len 36864 data: 9 8 8 9 6 5 5 6 6 29/30 Read block 55 len 40960 30/30 Read block 282 len 40960 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Write 10 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Read 9 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Read 10 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1002)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:19.777810121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:19.777995446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1520,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 96 96 94 155 155 152 T T F <--- 2 040-059 94 94 94 152 152 152 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 89 89 89 145 145 145 T T F <--- 5 100-119 81 81 81 135 135 135 T T F <--- 6 120-139 90 90 90 146 146 146 T T F <--- 7 140-159 94 94 94 152 152 152 T T F <--- 8 160-179 94 94 94 152 152 152 T T F <--- 9 180-199 94 94 94 152 152 152 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 95 95 95 154 154 154 F F F 12 240-259 95 95 95 154 154 154 T T F <--- 13 260-279 92 92 92 150 150 150 F F F 14 280-299 92 92 92 150 150 150 F F F 15 300-319 96 96 95 155 155 154 F F F <--- 16 320-339 83 83 83 139 139 139 F F F 17 340-359 87 87 87 143 143 143 T T F <--- 18 360-379 93 93 93 151 151 151 T T F <--- 19 380-399 91 91 91 148 148 148 T T F <--- 20 400-419 89 89 89 145 145 145 T T F <--- 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 94 94 94 152 152 152 F F F 25 500-519 94 94 94 152 152 152 F F F 26 520-539 92 92 92 150 150 150 T T F <--- 27 540-559 92 92 92 150 150 150 T T F <--- 28 560-579 90 90 90 146 146 146 T T F <--- 29 580-599 94 94 94 152 152 152 F F F Max gen: 96, Max flush: 155 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 94 94 94 152 152 152 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 97 97 97 156 156 156 F F F 5 100-119 97 97 97 156 156 156 F F F 6 120-139 97 97 97 156 156 156 F F F 7 140-159 97 97 97 156 156 156 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 94 94 94 152 152 152 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 95 95 95 154 154 154 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 92 92 92 150 150 150 F F F 15 300-319 96 96 96 155 155 155 F F F 16 320-339 83 83 83 139 139 139 F F F 17 340-359 97 97 97 156 156 156 F F F 18 360-379 97 97 97 156 156 156 F F F 19 380-399 97 97 97 156 156 156 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 94 94 94 152 152 152 F F F 25 500-519 94 94 94 152 152 152 F F F 26 520-539 97 97 97 156 156 156 F F F 27 540-559 97 97 97 156 156 156 F F F 28 560-579 97 97 97 156 156 156 F F F 29 580-599 94 94 94 152 152 152 F F F Max gen: 97, Max flush: 156 Begin loop 49 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.04235507Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.050479032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.050778823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.050803133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.050811882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.050823772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Crucible b4843004-32a2-4ee6-8aa0-ce043d215a81 has session id: 9ee4a412-5f6a-460a-823e-68c8b98b584a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051158253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Upstairs opts: Upstairs UUID: b4843004-32a2-4ee6-8aa0-ce043d215a81, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051174792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Crucible stats registered with UUID: b4843004-32a2-4ee6-8aa0-ce043d215a81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051184862Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:24.051196291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051350627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051367437Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051377596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051481513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051495273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"b4843004-32a2-4ee6-8aa0-ce043d215a81 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051527512Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051537242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051547092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051556911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051566691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051576991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.05161618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"ds_connection connected from Ok(127.0.0.1:57687)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051627399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44245)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051637989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"ds_connection connected from Ok(127.0.0.1:49625)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.051682118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052235162Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052253712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052357129Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052561233Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052570873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[0]R flush_numbers[0..12]: [152, 156, 152, 145, 156, 156, 156, 156, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.05265594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[0]R generation[0..12]: [94, 97, 94, 89, 97, 97, 97, 97, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.05266733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.05267841Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[1]R flush_numbers[0..12]: [152, 156, 152, 145, 156, 156, 156, 156, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.05268902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[1]R generation[0..12]: [94, 97, 94, 89, 97, 97, 97, 97, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052699349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052710459Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[2]R flush_numbers[0..12]: [152, 156, 152, 145, 156, 156, 156, 156, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052721249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[2]R generation[0..12]: [94, 97, 94, 89, 97, 97, 97, 97, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052731648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052742698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052754588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052764357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Next flush: 157","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052773657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052782977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"":"downstairs","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052792607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052801336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"0","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052811216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"1","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052820956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"client":"2","":"downstairs","session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052830926Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052840095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"b4843004-32a2-4ee6-8aa0-ce043d215a81 is now active with session: 9ee4a412-5f6a-460a-823e-68c8b98b584a","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052850275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052858995Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"session_id":"9ee4a412-5f6a-460a-823e-68c8b98b584a"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052867884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Activated sub_volume b4843004-32a2-4ee6-8aa0-ce043d215a81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052875904Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.052889614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.053167506Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} 01/30 Write block 376 len 28672 data: 10 11 8 7 7 8 7 02/30 Write block 148 len 24576 data: 9 12 11 13 12 10 03/30 Flush 04/30 Write block 391 len 24576 data: 6 9 10 9 10 10 05/30 Write block 20 len 40960 data: 10 10 12 10 9 9 9 8 7 7 06/30 Read block 100 len 24576 07/30 Write block 337 len 28672 data: 7 5 6 7 8 6 6 08/30 Read block 304 len 36864 09/30 Write block 516 len 28672 data: 9 9 11 12 12 10 9 10/30 Read block 201 len 28672 11/30 Read block 194 len 4096 12/30 Write block 41 len 24576 data: 13 14 14 14 12 14 13/30 Read block 400 len 20480 14/30 Write block 326 len 4096 data: 11 15/30 Read block 38 len 4096 16/30 Read block 415 len 32768 17/30 Read block 478 len 8192 18/30 Write block 297 len 20480 data: 6 7 6 8 7 19/30 Read block 111 len 28672 20/30 Write block 25 len 40960 data: 10 10 9 8 8 9 9 8 8 11 21/30 Write block 111 len 40960 data: 8 7 9 7 6 9 11 13 15 12 22/30 Read block 13 len 12288 23/30 Write block 531 len 16384 data: 10 10 8 8 24/30 Flush 25/30 Write block 599 len 4096 data: 3 26/30 Read block 335 len 40960 27/30 Read block 461 len 4096 28/30 Write block 490 len 8192 data: 7 7 29/30 Write block 105 len 24576 data: 4 4 5 7 8 7 30/30 Write block 247 len 20480 data: 9 12 13 15 14 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:24.061595269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:24.061794964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1528,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 98 98 97 158 158 156 F F F <--- 2 040-059 98 98 94 158 158 152 F F F <--- 3 060-079 89 89 89 145 145 145 F F F 4 080-099 97 97 97 156 156 156 F F F 5 100-119 98 98 97 158 158 156 T T F <--- 6 120-139 98 98 97 158 158 156 F F F <--- 7 140-159 98 98 97 157 157 156 F F T <--- 8 160-179 97 97 97 156 156 156 F F F 9 180-199 94 94 94 152 152 152 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 95 95 95 154 154 154 F F F 12 240-259 97 97 97 156 156 156 T T F <--- 13 260-279 92 92 92 150 150 150 F F F 14 280-299 98 98 92 158 158 150 F F F <--- 15 300-319 98 98 96 158 158 155 F F F <--- 16 320-339 98 98 83 158 158 139 F F F <--- 17 340-359 98 98 97 158 158 156 F F F <--- 18 360-379 98 98 97 157 157 156 F F T <--- 19 380-399 98 98 97 158 158 156 F F T <--- 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 94 94 94 152 152 152 T T F <--- 25 500-519 98 98 94 158 158 152 F F F <--- 26 520-539 98 98 97 158 158 156 F F F <--- 27 540-559 97 97 97 156 156 156 F F F 28 560-579 97 97 97 156 156 156 F F F 29 580-599 94 94 94 152 152 152 T T F <--- Max gen: 98, Max flush: 158 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 98 98 98 158 158 158 F F F 2 040-059 98 98 98 158 158 158 F F F 3 060-079 89 89 89 145 145 145 F F F 4 080-099 97 97 97 156 156 156 F F F 5 100-119 99 99 99 159 159 159 F F F 6 120-139 98 98 98 158 158 158 F F F 7 140-159 98 98 98 157 157 157 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 94 94 94 152 152 152 F F F 10 200-219 89 89 89 145 145 145 F F F 11 220-239 95 95 95 154 154 154 F F F 12 240-259 99 99 99 159 159 159 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 98 98 98 158 158 158 F F F 15 300-319 98 98 98 158 158 158 F F F 16 320-339 98 98 98 158 158 158 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 98 98 157 157 157 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 F F F 23 460-479 89 89 89 145 145 145 F F F 24 480-499 99 99 99 159 159 159 F F F 25 500-519 98 98 98 158 158 158 F F F 26 520-539 98 98 98 158 158 158 F F F 27 540-559 97 97 97 156 156 156 F F F 28 560-579 97 97 97 156 156 156 F F F 29 580-599 99 99 99 159 159 159 F F F Max gen: 99, Max flush: 159 Begin loop 50 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.325459458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.333689007Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.333974658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.333999948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334008618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334020157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Crucible ae324680-f6ee-4f78-8615-dbc1be7372ae has session id: b8f50e26-2acb-4ef6-828a-19ca58a2513b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334355158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Upstairs opts: Upstairs UUID: ae324680-f6ee-4f78-8615-dbc1be7372ae, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334378827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Crucible stats registered with UUID: ae324680-f6ee-4f78-8615-dbc1be7372ae","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334388607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:28.334397547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334550912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334568032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334578092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334709858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334720808Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"ae324680-f6ee-4f78-8615-dbc1be7372ae active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334758876Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334768286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334778106Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334787666Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334797135Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334807205Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334816985Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"ds_connection connected from Ok(127.0.0.1:57446)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334875473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"ds_connection connected from Ok(127.0.0.1:52449)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334900912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"ds_connection connected from Ok(127.0.0.1:44019)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.334946401Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335480276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335502416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335564314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335795107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335814287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[0]R flush_numbers[0..12]: [152, 158, 158, 145, 156, 159, 158, 157, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335899604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[0]R generation[0..12]: [94, 98, 98, 89, 97, 99, 98, 98, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335910674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335921914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[1]R flush_numbers[0..12]: [152, 158, 158, 145, 156, 159, 158, 157, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335932693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[1]R generation[0..12]: [94, 98, 98, 89, 97, 99, 98, 98, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335943113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335953893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[2]R flush_numbers[0..12]: [152, 158, 158, 145, 156, 159, 158, 157, 156, 152, 145, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335964553Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[2]R generation[0..12]: [94, 98, 98, 89, 97, 99, 98, 98, 97, 94, 89, 95]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335974862Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335985762Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.335995142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336004901Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Next flush: 160","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336014381Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336023741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"":"downstairs","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336033471Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.33604404Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"0","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.33605383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"1","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.33606342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"client":"2","":"downstairs","session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336073169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336082299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"ae324680-f6ee-4f78-8615-dbc1be7372ae is now active with session: b8f50e26-2acb-4ef6-828a-19ca58a2513b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336092239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336100899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"session_id":"b8f50e26-2acb-4ef6-828a-19ca58a2513b"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336109918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Activated sub_volume ae324680-f6ee-4f78-8615-dbc1be7372ae","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336117758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336125818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.336478198Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} 01/30 Write block 123 len 4096 data: 12 02/30 Write block 310 len 40960 data: 8 9 9 9 8 7 7 8 7 7 03/30 Read block 335 len 20480 04/30 Flush 05/30 Write block 301 len 40960 data: 8 6 6 7 6 8 8 8 8 9 06/30 Write block 155 len 20480 data: 8 8 5 4 5 07/30 Write block 513 len 20480 data: 8 9 8 10 10 08/30 Read block 279 len 24576 09/30 Read block 260 len 28672 10/30 Read block 177 len 36864 11/30 Read block 393 len 20480 12/30 Write block 27 len 40960 data: 10 9 9 10 10 9 9 12 10 9 13/30 Read block 185 len 4096 14/30 Write block 225 len 8192 data: 10 10 15/30 Write block 470 len 28672 data: 5 5 7 7 8 8 9 16/30 Write block 20 len 4096 data: 11 17/30 Read block 438 len 36864 18/30 Read block 392 len 12288 19/30 Write block 84 len 16384 data: 7 6 6 8 20/30 Write block 56 len 40960 data: 10 11 9 10 7 6 5 6 5 7 21/30 Write block 215 len 20480 data: 6 7 5 6 6 22/30 Write block 326 len 20480 data: 12 11 10 12 9 23/30 Flush 24/30 Write block 181 len 28672 data: 8 8 9 10 9 10 11 25/30 Flush 26/30 Write block 107 len 28672 data: 6 8 9 8 9 8 10 27/30 Read block 60 len 12288 28/30 Write block 590 len 28672 data: 7 7 7 5 6 4 4 29/30 Read block 513 len 8192 30/30 Write block 442 len 28672 data: 7 7 10 13 11 10 9 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 29 29 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:28.345418767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:28.345612251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1536,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 98 161 161 158 F F F <--- 2 040-059 100 100 98 161 161 158 F F F <--- 3 060-079 100 100 89 161 161 145 F F F <--- 4 080-099 100 100 97 161 161 156 F F F <--- 5 100-119 99 99 99 159 159 159 T T F <--- 6 120-139 100 100 98 160 160 158 F F T <--- 7 140-159 100 100 98 161 161 157 F F F <--- 8 160-179 97 97 97 156 156 156 F F F 9 180-199 100 100 94 162 162 152 F F F <--- 10 200-219 100 100 89 161 161 145 F F F <--- 11 220-239 100 100 95 161 161 154 F F F <--- 12 240-259 99 99 99 159 159 159 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 98 98 98 158 158 158 F F F 15 300-319 100 100 98 161 161 158 F F T <--- 16 320-339 100 100 98 161 161 158 F F F <--- 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 98 98 157 157 157 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 94 94 94 153 153 153 T T F <--- 23 460-479 100 100 89 161 161 145 F F F <--- 24 480-499 99 99 99 159 159 159 F F F 25 500-519 100 100 98 161 161 158 F F F <--- 26 520-539 98 98 98 158 158 158 F F F 27 540-559 97 97 97 156 156 156 F F F 28 560-579 97 97 97 156 156 156 F F F 29 580-599 99 99 99 159 159 159 T T F <--- Max gen: 100, Max flush: 162 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 100 100 100 161 161 161 F F F 3 060-079 100 100 100 161 161 161 F F F 4 080-099 100 100 100 161 161 161 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 100 100 100 161 161 161 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 100 100 100 162 162 162 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 100 100 100 161 161 161 F F F 12 240-259 99 99 99 159 159 159 F F F 13 260-279 92 92 92 150 150 150 F F F 14 280-299 98 98 98 158 158 158 F F F 15 300-319 100 100 100 161 161 161 F F F 16 320-339 100 100 100 161 161 161 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 98 98 157 157 157 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F F F 22 440-459 101 101 101 163 163 163 F F F 23 460-479 100 100 100 161 161 161 F F F 24 480-499 99 99 99 159 159 159 F F F 25 500-519 100 100 100 161 161 161 F F F 26 520-539 98 98 98 158 158 158 F F F 27 540-559 97 97 97 156 156 156 F F F 28 560-579 97 97 97 156 156 156 F F F 29 580-599 101 101 101 163 163 163 F F F Max gen: 101, Max flush: 163 Begin loop 51 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.611540812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:32.61874021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.618796288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.625737503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626010555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626036235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626047414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626059174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Crucible 3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd has session id: 97bd170a-9ec8-45d2-9dbc-344b6eee9c3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626372945Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Upstairs opts: Upstairs UUID: 3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626415894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Crucible stats registered with UUID: 3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626426404Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:32.626435363Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626589799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626607118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626617218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626740325Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626763164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626804343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626814013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626824482Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626836782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626846502Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626856421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626866231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:65012)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.626981418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:57976)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627006597Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63150)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627050146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627649329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627688628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627741767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.62798773Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.627997739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[0]R flush_numbers[0..12]: [152, 161, 161, 161, 161, 163, 160, 161, 156, 162, 161, 161]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628058018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[0]R generation[0..12]: [94, 100, 100, 100, 100, 101, 100, 100, 97, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628069527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628080697Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[1]R flush_numbers[0..12]: [152, 161, 161, 161, 161, 163, 160, 161, 156, 162, 161, 161]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628093887Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[1]R generation[0..12]: [94, 100, 100, 100, 100, 101, 100, 100, 97, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628104676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628115646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[2]R flush_numbers[0..12]: [152, 161, 161, 161, 161, 163, 160, 161, 156, 162, 161, 161]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628126446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[2]R generation[0..12]: [94, 100, 100, 100, 100, 101, 100, 100, 97, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628137095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628148095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628157525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628167545Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628177184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628186614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"":"downstairs","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628196654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628205594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"0","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628215843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"1","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628225813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"client":"2","":"downstairs","session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628237263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628246372Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd is now active with session: 97bd170a-9ec8-45d2-9dbc-344b6eee9c3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628256532Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628265362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"session_id":"97bd170a-9ec8-45d2-9dbc-344b6eee9c3e"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628274672Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Activated sub_volume 3c6da8d6-bf8f-4c61-a4aa-3dd4868a10bd","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628282781Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628290911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.628636221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} 01/30 Write block 246 len 20480 data: 9 10 13 14 16 02/30 Read block 468 len 16384 03/30 Write block 573 len 8192 data: 5 5 04/30 Read block 489 len 12288 05/30 Write block 41 len 16384 data: 14 15 15 15 06/30 Write block 426 len 16384 data: 11 8 7 8 07/30 Read block 523 len 36864 08/30 Write block 214 len 40960 data: 6 7 8 6 7 7 8 10 8 8 09/30 Read block 505 len 24576 10/30 Write block 508 len 8192 data: 5 5 11/30 Read block 325 len 20480 12/30 Write block 306 len 16384 data: 9 9 9 9 13/30 Read block 184 len 36864 14/30 Write block 283 len 28672 data: 6 4 4 5 5 4 3 15/30 Read block 62 len 12288 16/30 Read block 463 len 32768 17/30 Write block 482 len 4096 data: 7 18/30 Read block 304 len 32768 19/30 Read block 398 len 12288 20/30 Read block 65 len 8192 21/30 Read block 240 len 40960 22/30 Read block 266 len 16384 23/30 Read block 399 len 24576 24/30 Write block 548 len 40960 data: 8 8 11 11 13 13 10 10 11 10 25/30 Read block 219 len 28672 26/30 Write block 470 len 32768 data: 6 6 8 8 9 9 10 6 27/30 Write block 53 len 12288 data: 10 9 8 28/30 Write block 318 len 8192 data: 8 8 29/30 Read block 142 len 12288 30/30 Write block 265 len 40960 data: 9 7 8 9 12 14 10 10 10 10 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:32.637322607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:32.637487883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1544,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 100 100 100 161 161 161 F T T <--- 3 060-079 100 100 100 161 161 161 F F F 4 080-099 100 100 100 161 161 161 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 100 100 100 161 161 161 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 100 100 100 162 162 162 F F F 10 200-219 100 100 100 161 161 161 F T T <--- 11 220-239 100 100 100 161 161 161 F T T <--- 12 240-259 99 99 99 159 159 159 T T T 13 260-279 92 92 92 150 150 150 F T T <--- 14 280-299 98 98 98 158 158 158 F T T <--- 15 300-319 100 100 100 161 161 161 F T T <--- 16 320-339 100 100 100 161 161 161 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 98 98 157 157 157 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 92 92 92 150 150 150 F T T <--- 22 440-459 101 101 101 163 163 163 F F F 23 460-479 100 100 100 161 161 161 F T T <--- 24 480-499 99 99 99 159 159 159 F T T <--- 25 500-519 100 100 100 161 161 161 F T T <--- 26 520-539 98 98 98 158 158 158 F F F 27 540-559 97 97 97 156 156 156 F T T <--- 28 560-579 97 97 97 156 156 156 T T T 29 580-599 101 101 101 163 163 163 F F F Max gen: 101, Max flush: 163 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 102 102 102 164 164 164 F F F 3 060-079 100 100 100 161 161 161 F F F 4 080-099 100 100 100 161 161 161 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 100 100 100 161 161 161 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 100 100 100 162 162 162 F F F 10 200-219 102 102 102 164 164 164 F F F 11 220-239 102 102 102 164 164 164 F F F 12 240-259 102 102 102 164 164 164 F F F 13 260-279 102 102 102 164 164 164 F F F 14 280-299 102 102 102 164 164 164 F F F 15 300-319 102 102 102 164 164 164 F F F 16 320-339 100 100 100 161 161 161 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 98 98 157 157 157 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 102 102 102 164 164 164 F F F 22 440-459 101 101 101 163 163 163 F F F 23 460-479 102 102 102 164 164 164 F F F 24 480-499 102 102 102 164 164 164 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 98 98 98 158 158 158 F F F 27 540-559 102 102 102 164 164 164 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 101 101 101 163 163 163 F F F Max gen: 102, Max flush: 164 Begin loop 52 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.909633828Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:36.916818226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.916861845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.923716302Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.923988694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924014134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924022863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924034533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Crucible 7d4984eb-eef7-411d-aebf-8641c7399c9f has session id: d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924356064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Upstairs opts: Upstairs UUID: 7d4984eb-eef7-411d-aebf-8641c7399c9f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924382103Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Crucible stats registered with UUID: 7d4984eb-eef7-411d-aebf-8641c7399c9f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924392123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:36.924400973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924563168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924579998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924590237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924710884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924725024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"7d4984eb-eef7-411d-aebf-8641c7399c9f active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924762603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924772092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924781912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924791842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924801581Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924811681Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.924823291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"ds_connection connected from Ok(127.0.0.1:40769)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92485856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"ds_connection connected from Ok(127.0.0.1:65055)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92486981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"ds_connection connected from Ok(127.0.0.1:47251)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925003776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925602249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925622238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925747665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925950739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.925960919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[0]R flush_numbers[0..12]: [152, 161, 164, 161, 161, 163, 160, 161, 156, 162, 164, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926103285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[0]R generation[0..12]: [94, 100, 102, 100, 100, 101, 100, 100, 97, 100, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926129304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926141294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[1]R flush_numbers[0..12]: [152, 161, 164, 161, 161, 163, 160, 161, 156, 162, 164, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926152164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[1]R generation[0..12]: [94, 100, 102, 100, 100, 101, 100, 100, 97, 100, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926163093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926176133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[2]R flush_numbers[0..12]: [152, 161, 164, 161, 161, 163, 160, 161, 156, 162, 164, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926186983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[2]R generation[0..12]: [94, 100, 102, 100, 100, 101, 100, 100, 97, 100, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926197442Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926208452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Max found gen is 103","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926217912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Generation requested: 104 >= found:103","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926227861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Next flush: 165","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926237151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926246541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"":"downstairs","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926256311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92626511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"0","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92627516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"1","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92628482Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"client":"2","":"downstairs","session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.92629462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926303869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"7d4984eb-eef7-411d-aebf-8641c7399c9f is now active with session: d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926314059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926322899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"session_id":"d63ff8c7-ab0c-469c-bcdd-4da5b3114eb9"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926333638Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Activated sub_volume 7d4984eb-eef7-411d-aebf-8641c7399c9f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926341688Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Wait for a query_work_queue command to finish before sending IO"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926354568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.926655289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} 01/30 Write block 238 len 40960 data: 10 11 11 9 8 7 7 6 10 11 02/30 Read block 111 len 36864 03/30 Read block 17 len 24576 04/30 Flush 05/30 Write block 304 len 36864 data: 8 7 10 10 10 10 10 10 10 06/30 Read block 190 len 16384 07/30 Flush 08/30 Write block 481 len 12288 data: 4 8 6 09/30 Write block 436 len 8192 data: 8 7 10/30 Write block 92 len 32768 data: 5 5 8 8 8 6 4 5 11/30 Read block 546 len 4096 12/30 Write block 150 len 20480 data: 12 14 13 11 7 13/30 Write block 58 len 28672 data: 10 11 8 7 6 7 6 14/30 Read block 2 len 16384 15/30 Write block 365 len 20480 data: 11 9 9 10 9 16/30 Write block 402 len 8192 data: 9 7 17/30 Write block 478 len 40960 data: 4 4 4 5 9 7 6 5 4 5 18/30 Read block 586 len 16384 19/30 Read block 429 len 28672 20/30 Read block 541 len 40960 21/30 Read block 569 len 28672 22/30 Read block 550 len 24576 23/30 Read block 95 len 28672 24/30 Flush 25/30 Write block 184 len 28672 data: 11 10 11 12 11 10 9 26/30 Read block 44 len 4096 27/30 Read block 214 len 4096 28/30 Write block 581 len 28672 data: 9 12 11 11 10 9 8 29/30 Read block 374 len 20480 30/30 Read block 131 len 16384 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:36.936468774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:36.936651818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1552,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 102 104 104 164 167 167 F F F <--- 3 060-079 100 104 104 161 167 167 F F F <--- 4 080-099 100 104 104 161 167 167 F F F <--- 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 100 104 104 161 167 167 F F F <--- 8 160-179 97 97 97 156 156 156 F F F 9 180-199 100 100 100 162 162 162 F T T <--- 10 200-219 102 102 102 164 164 164 F F F 11 220-239 102 104 104 164 165 165 F F F <--- 12 240-259 102 104 104 164 165 165 F F F <--- 13 260-279 102 102 102 164 164 164 F F F 14 280-299 102 102 102 164 164 164 F F F 15 300-319 102 104 104 164 166 166 F F F <--- 16 320-339 100 100 100 161 161 161 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 98 104 104 157 167 167 F F F <--- 19 380-399 98 98 98 158 158 158 F F F 20 400-419 97 104 104 156 167 167 F F F <--- 21 420-439 102 104 104 164 167 167 F F F <--- 22 440-459 101 101 101 163 163 163 F F F 23 460-479 102 104 104 164 167 167 F F F <--- 24 480-499 102 104 104 164 167 167 F F F <--- 25 500-519 102 102 102 164 164 164 F F F 26 520-539 98 98 98 158 158 158 F F F 27 540-559 102 102 102 164 164 164 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 101 101 101 163 163 163 F T T <--- Max gen: 104, Max flush: 167 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 104 104 104 167 167 167 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 104 104 104 167 167 167 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 104 104 104 167 167 167 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 105 105 105 168 168 168 F F F 10 200-219 102 102 102 164 164 164 F F F 11 220-239 104 104 104 165 165 165 F F F 12 240-259 104 104 104 165 165 165 F F F 13 260-279 102 102 102 164 164 164 F F F 14 280-299 102 102 102 164 164 164 F F F 15 300-319 104 104 104 166 166 166 F F F 16 320-339 100 100 100 161 161 161 F F F 17 340-359 98 98 98 158 158 158 F F F 18 360-379 104 104 104 167 167 167 F F F 19 380-399 98 98 98 158 158 158 F F F 20 400-419 104 104 104 167 167 167 F F F 21 420-439 104 104 104 167 167 167 F F F 22 440-459 101 101 101 163 163 163 F F F 23 460-479 104 104 104 167 167 167 F F F 24 480-499 104 104 104 167 167 167 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 98 98 98 158 158 158 F F F 27 540-559 102 102 102 164 164 164 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 105 105 105 168 168 168 F F F Max gen: 105, Max flush: 168 Begin loop 53 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.204270232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:41.211398971Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.21144661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.21821086Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218476642Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218506391Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218514981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218526731Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Crucible cae14d38-53b5-4011-8b05-e5cb074e3f65 has session id: c01622d7-a10c-4887-adbe-30b4195cab10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218832902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Upstairs opts: Upstairs UUID: cae14d38-53b5-4011-8b05-e5cb074e3f65, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218866321Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Crucible stats registered with UUID: cae14d38-53b5-4011-8b05-e5cb074e3f65","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.218879511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:41.218888481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219032057Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219048016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219058256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219200232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219213981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"cae14d38-53b5-4011-8b05-e5cb074e3f65 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219237011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219246211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.21925594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.21926576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.21927515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219285039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219321138Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"ds_connection connected from Ok(127.0.0.1:50598)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219341958Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"ds_connection connected from Ok(127.0.0.1:44151)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219352838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"ds_connection connected from Ok(127.0.0.1:60352)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.219403536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220085757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220104996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220141115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220380929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220415458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[0]R flush_numbers[0..12]: [152, 161, 167, 167, 167, 163, 160, 167, 156, 168, 164, 165]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220509495Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[0]R generation[0..12]: [94, 100, 104, 104, 104, 101, 100, 104, 97, 105, 102, 104]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220520645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220531544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[1]R flush_numbers[0..12]: [152, 161, 167, 167, 167, 163, 160, 167, 156, 168, 164, 165]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220542104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[1]R generation[0..12]: [94, 100, 104, 104, 104, 101, 100, 104, 97, 105, 102, 104]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220552544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220563294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[2]R flush_numbers[0..12]: [152, 161, 167, 167, 167, 163, 160, 167, 156, 168, 164, 165]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220573783Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[2]R generation[0..12]: [94, 100, 104, 104, 104, 101, 100, 104, 97, 105, 102, 104]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220584223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220597393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220606772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220616582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Next flush: 169","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220625922Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220635141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"":"downstairs","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220644801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220653601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"0","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220663291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"1","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.22067299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"client":"2","":"downstairs","session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.22068322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.22069256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"cae14d38-53b5-4011-8b05-e5cb074e3f65 is now active with session: c01622d7-a10c-4887-adbe-30b4195cab10","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.22070249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220711119Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"session_id":"c01622d7-a10c-4887-adbe-30b4195cab10"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220720269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Activated sub_volume cae14d38-53b5-4011-8b05-e5cb074e3f65","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220728229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.220736289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.221098758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} 01/30 Write block 520 len 12288 data: 13 11 10 02/30 Read block 402 len 20480 03/30 Read block 459 len 16384 04/30 Write block 81 len 20480 data: 7 7 7 8 7 05/30 Flush 06/30 Write block 532 len 40960 data: 11 9 9 7 9 11 9 7 9 8 07/30 Write block 441 len 32768 data: 7 8 8 11 14 12 11 10 08/30 Write block 233 len 4096 data: 6 09/30 Write block 161 len 8192 data: 9 10 10/30 Write block 388 len 32768 data: 10 10 8 7 10 11 10 11 11/30 Read block 179 len 16384 12/30 Write block 199 len 28672 data: 13 10 10 9 10 9 8 13/30 Read block 429 len 16384 14/30 Flush 15/30 Write block 547 len 32768 data: 8 9 9 12 12 14 14 11 16/30 Read block 433 len 24576 17/30 Flush 18/30 Write block 389 len 16384 data: 11 9 8 11 19/30 Flush 20/30 Write block 339 len 32768 data: 7 8 9 7 7 6 6 7 21/30 Read block 131 len 16384 22/30 Write block 584 len 4096 data: 12 23/30 Write block 141 len 40960 data: 8 7 7 9 11 9 6 10 13 13 24/30 Read block 574 len 28672 25/30 Read block 352 len 28672 26/30 Read block 262 len 24576 27/30 Write block 251 len 16384 data: 15 12 11 11 28/30 Read block 296 len 20480 29/30 Write block 396 len 20480 data: 11 10 9 7 8 30/30 Write block 65 len 16384 data: 8 8 7 6 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 10 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 0 23 23 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:41.229776725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:41.229961489Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1560,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 104 104 104 167 167 167 F F F 3 060-079 104 104 104 167 167 167 F T T <--- 4 080-099 106 106 106 169 169 169 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 104 104 104 167 167 167 F T T <--- 8 160-179 97 106 106 156 170 170 F F F <--- 9 180-199 105 106 106 168 170 170 F F F <--- 10 200-219 102 106 106 164 170 170 F F F <--- 11 220-239 104 106 106 165 170 170 F F F <--- 12 240-259 104 104 104 165 165 165 F T T <--- 13 260-279 102 102 102 164 164 164 F F F 14 280-299 102 102 102 164 164 164 F F F 15 300-319 104 104 104 166 166 166 F F F 16 320-339 100 100 100 161 161 161 F T T <--- 17 340-359 98 98 98 158 158 158 F T T <--- 18 360-379 104 104 104 167 167 167 F F F 19 380-399 98 106 106 158 172 172 F T T <--- 20 400-419 104 104 104 167 167 167 F T T <--- 21 420-439 104 104 104 167 167 167 F F F 22 440-459 101 106 106 163 170 170 F F F <--- 23 460-479 104 104 104 167 167 167 F F F 24 480-499 104 104 104 167 167 167 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 106 106 106 169 170 170 F F F <--- 27 540-559 102 106 106 164 171 171 F F F <--- 28 560-579 102 102 102 164 164 164 F F F 29 580-599 105 105 105 168 168 168 F T T <--- Max gen: 106, Max flush: 172 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 104 104 104 167 167 167 F F F 3 060-079 107 107 107 173 173 173 F F F 4 080-099 106 106 106 169 169 169 F F F 5 100-119 101 101 101 163 163 163 F F F 6 120-139 100 100 100 160 160 160 F F F 7 140-159 107 107 107 173 173 173 F F F 8 160-179 106 106 106 170 170 170 F F F 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 107 107 107 173 173 173 F F F 13 260-279 102 102 102 164 164 164 F F F 14 280-299 102 102 102 164 164 164 F F F 15 300-319 104 104 104 166 166 166 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 107 107 107 173 173 173 F F F 18 360-379 104 104 104 167 167 167 F F F 19 380-399 107 107 107 173 173 173 F F F 20 400-419 107 107 107 173 173 173 F F F 21 420-439 104 104 104 167 167 167 F F F 22 440-459 106 106 106 170 170 170 F F F 23 460-479 104 104 104 167 167 167 F F F 24 480-499 104 104 104 167 167 167 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 106 106 106 170 170 170 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 107, Max flush: 173 Begin loop 54 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.495784513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:45.502969261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.503025679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.509853257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.51012611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510150939Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510159589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510171409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Crucible 8ad0a006-5f36-4b44-bb5d-fd33b9e59446 has session id: e1e8a209-8e26-42e0-92ae-cc0b342a332e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510502199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Upstairs opts: Upstairs UUID: 8ad0a006-5f36-4b44-bb5d-fd33b9e59446, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510534188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Crucible stats registered with UUID: 8ad0a006-5f36-4b44-bb5d-fd33b9e59446","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510544748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:45.510553798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510710113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510744292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510757772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510882409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510900678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"8ad0a006-5f36-4b44-bb5d-fd33b9e59446 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510918768Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510927757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510937687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510947717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510957536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510967936Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.510977986Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"ds_connection connected from Ok(127.0.0.1:49383)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.511031874Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"ds_connection connected from Ok(127.0.0.1:49444)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.511042974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"ds_connection connected from Ok(127.0.0.1:40888)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.511053334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.511801663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.511857621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.51191216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512135083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512145913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[0]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 163, 160, 173, 170, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512217431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[0]R generation[0..12]: [94, 100, 104, 107, 106, 101, 100, 107, 106, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512228731Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.51223994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[1]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 163, 160, 173, 170, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.51225064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[1]R generation[0..12]: [94, 100, 104, 107, 106, 101, 100, 107, 106, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.51226109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512271879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[2]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 163, 160, 173, 170, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512282599Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[2]R generation[0..12]: [94, 100, 104, 107, 106, 101, 100, 107, 106, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512293199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512304039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512313498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512323278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Next flush: 174","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512332788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512343957Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"":"downstairs","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512353857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512362757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"0","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512372627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"1","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512382546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"client":"2","":"downstairs","session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512392646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512401546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"8ad0a006-5f36-4b44-bb5d-fd33b9e59446 is now active with session: e1e8a209-8e26-42e0-92ae-cc0b342a332e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512411556Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512420335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"session_id":"e1e8a209-8e26-42e0-92ae-cc0b342a332e"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512429555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Activated sub_volume 8ad0a006-5f36-4b44-bb5d-fd33b9e59446","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512437445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512445425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.512790045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} 01/30 Write block 483 len 40960 data: 8 7 6 5 6 6 6 8 8 8 02/30 Read block 237 len 28672 03/30 Write block 114 len 16384 data: 8 7 10 12 04/30 Read block 394 len 24576 05/30 Read block 171 len 8192 06/30 Write block 268 len 16384 data: 10 13 15 11 07/30 Write block 170 len 32768 data: 6 6 7 7 7 7 7 8 08/30 Flush 09/30 Write block 279 len 20480 data: 10 9 8 8 7 10/30 Read block 504 len 28672 11/30 Read block 169 len 12288 12/30 Read block 284 len 4096 13/30 Read block 222 len 36864 14/30 Read block 204 len 16384 15/30 Read block 130 len 16384 16/30 Write block 494 len 8192 data: 8 9 17/30 Read block 531 len 36864 18/30 Write block 433 len 8192 data: 7 8 19/30 Read block 0 len 32768 20/30 Write block 118 len 20480 data: 14 16 13 14 12 21/30 Write block 312 len 16384 data: 11 10 9 8 22/30 Write block 343 len 8192 data: 8 7 23/30 Write block 361 len 4096 data: 7 24/30 Read block 34 len 32768 25/30 Read block 140 len 36864 26/30 Read block 266 len 28672 27/30 Write block 125 len 20480 data: 11 10 10 7 5 28/30 Read block 531 len 24576 29/30 Write block 466 len 36864 data: 4 5 4 4 7 7 9 9 10 30/30 Read block 439 len 16384 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 9 Sent Sent Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 2 0 29 Done 3 28 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1007)) Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:45.521426472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:45.521609417Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1568,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 104 104 104 167 167 167 F F F 3 060-079 107 107 107 173 173 173 F F F 4 080-099 106 106 106 169 169 169 F F F 5 100-119 101 108 108 163 174 174 T T T <--- 6 120-139 100 100 100 160 160 160 F T T <--- 7 140-159 107 107 107 173 173 173 F F F 8 160-179 106 108 108 170 174 174 F F F <--- 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 107 107 107 173 173 173 F F F 13 260-279 102 108 108 164 174 174 F T T <--- 14 280-299 102 102 102 164 164 164 F T T <--- 15 300-319 104 104 104 166 166 166 F T T <--- 16 320-339 107 107 107 173 173 173 F F F 17 340-359 107 107 107 173 173 173 F T T <--- 18 360-379 104 104 104 167 167 167 F T T <--- 19 380-399 107 107 107 173 173 173 F F F 20 400-419 107 107 107 173 173 173 F F F 21 420-439 104 104 104 167 167 167 F T T <--- 22 440-459 106 106 106 170 170 170 F F F 23 460-479 104 104 104 167 167 167 F T T <--- 24 480-499 104 108 108 167 174 174 T T T <--- 25 500-519 102 102 102 164 164 164 F F F 26 520-539 106 106 106 170 170 170 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 108, Max flush: 174 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 F F F 2 040-059 104 104 104 167 167 167 F F F 3 060-079 107 107 107 173 173 173 F F F 4 080-099 106 106 106 169 169 169 F F F 5 100-119 109 109 109 175 175 175 F F F 6 120-139 109 109 109 175 175 175 F F F 7 140-159 107 107 107 173 173 173 F F F 8 160-179 108 108 108 174 174 174 F F F 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 107 107 107 173 173 173 F F F 13 260-279 109 109 109 175 175 175 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 109 109 109 175 175 175 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 109 109 109 175 175 175 F F F 18 360-379 109 109 109 175 175 175 F F F 19 380-399 107 107 107 173 173 173 F F F 20 400-419 107 107 107 173 173 173 F F F 21 420-439 109 109 109 175 175 175 F F F 22 440-459 106 106 106 170 170 170 F F F 23 460-479 109 109 109 175 175 175 F F F 24 480-499 109 109 109 175 175 175 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 106 106 106 170 170 170 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 109, Max flush: 175 Begin loop 55 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.785994831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79422713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.794525431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.794550861Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79456189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79457357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Crucible c6602b2d-f456-4c55-a28d-ec4f7920e78d has session id: 02bc4f25-d2ec-4e2a-8401-56aa64708202","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.794892151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Upstairs opts: Upstairs UUID: c6602b2d-f456-4c55-a28d-ec4f7920e78d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79492704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Crucible stats registered with UUID: c6602b2d-f456-4c55-a28d-ec4f7920e78d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.794952579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:49.794962939Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795115115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795132774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795142774Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79528785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795311909Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"c6602b2d-f456-4c55-a28d-ec4f7920e78d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795324279Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795333189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795342968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795352588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795364228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795374247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795408546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"ds_connection connected from Ok(127.0.0.1:46128)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795429596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"ds_connection connected from Ok(127.0.0.1:41985)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795440726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"ds_connection connected from Ok(127.0.0.1:49631)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.795563032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796126066Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796154765Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796219384Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796447947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796463977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[0]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 175, 175, 173, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796521665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[0]R generation[0..12]: [94, 100, 104, 107, 106, 109, 109, 107, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796533065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796544165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[1]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 175, 175, 173, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796557494Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[1]R generation[0..12]: [94, 100, 104, 107, 106, 109, 109, 107, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796568184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796579204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[2]R flush_numbers[0..12]: [152, 161, 167, 173, 169, 175, 175, 173, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796589823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[2]R generation[0..12]: [94, 100, 104, 107, 106, 109, 109, 107, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796600293Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796611263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796620592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796630242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Next flush: 176","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796639612Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796649112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"":"downstairs","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796659111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796668021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"0","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796677951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"1","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79668769Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"client":"2","":"downstairs","session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79669793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79670863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"c6602b2d-f456-4c55-a28d-ec4f7920e78d is now active with session: 02bc4f25-d2ec-4e2a-8401-56aa64708202","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.79671859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796727659Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796735289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"session_id":"02bc4f25-d2ec-4e2a-8401-56aa64708202"} {"msg":"Activated sub_volume c6602b2d-f456-4c55-a28d-ec4f7920e78d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796744509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.796752709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.797099159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} 01/30 Write block 242 len 36864 data: 9 8 8 7 11 12 14 15 17 02/30 Read block 575 len 24576 03/30 Read block 549 len 12288 04/30 Write block 104 len 40960 data: 6 5 5 7 9 10 9 10 9 11 05/30 Write block 412 len 12288 data: 7 6 6 06/30 Write block 340 len 4096 data: 9 07/30 Read block 399 len 12288 08/30 Read block 180 len 8192 09/30 Read block 30 len 8192 10/30 Read block 259 len 24576 11/30 Write block 37 len 8192 data: 10 12 12/30 Write block 153 len 20480 data: 12 8 9 9 6 13/30 Read block 408 len 8192 14/30 Write block 418 len 16384 data: 7 7 9 7 15/30 Read block 289 len 40960 16/30 Read block 397 len 36864 17/30 Read block 259 len 24576 18/30 Read block 431 len 16384 19/30 Write block 42 len 28672 data: 16 16 16 13 15 17 12 20/30 Read block 482 len 40960 21/30 Read block 411 len 36864 22/30 Read block 183 len 24576 23/30 Read block 468 len 24576 24/30 Write block 525 len 4096 data: 9 25/30 Read block 558 len 28672 26/30 Read block 46 len 40960 27/30 Read block 171 len 32768 28/30 Write block 360 len 32768 data: 7 8 5 6 7 12 10 10 29/30 Write block 42 len 40960 data: 17 17 17 14 16 18 13 11 12 11 30/30 Write block 312 len 4096 data: 12 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 1 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 23 3 29 Done 27 7 27 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:49.805305628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:49.805454424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1576,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 100 100 100 161 161 161 T F T <--- 2 040-059 104 104 104 167 167 167 T F T <--- 3 060-079 107 107 107 173 173 173 F F F 4 080-099 106 106 106 169 169 169 F F F 5 100-119 109 109 109 175 175 175 T T T 6 120-139 109 109 109 175 175 175 F F F 7 140-159 107 107 107 173 173 173 T F T <--- 8 160-179 108 108 108 174 174 174 F F F 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 107 107 107 173 173 173 T T T 13 260-279 109 109 109 175 175 175 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 109 109 109 175 175 175 T F T <--- 16 320-339 107 107 107 173 173 173 F F F 17 340-359 109 109 109 175 175 175 T T T 18 360-379 109 109 109 175 175 175 T F T <--- 19 380-399 107 107 107 173 173 173 F F F 20 400-419 107 107 107 173 173 173 T T T 21 420-439 109 109 109 175 175 175 T F T <--- 22 440-459 106 106 106 170 170 170 F F F 23 460-479 109 109 109 175 175 175 F F F 24 480-499 109 109 109 175 175 175 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 106 106 106 170 170 170 T F T <--- 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 109, Max flush: 175 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 152 152 152 F F F 1 020-039 110 110 110 176 176 176 F F F 2 040-059 110 110 110 176 176 176 F F F 3 060-079 107 107 107 173 173 173 F F F 4 080-099 106 106 106 169 169 169 F F F 5 100-119 110 110 110 176 176 176 F F F 6 120-139 109 109 109 175 175 175 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 108 108 108 174 174 174 F F F 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 110 110 110 176 176 176 F F F 13 260-279 109 109 109 175 175 175 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 110 110 110 176 176 176 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 110 110 110 176 176 176 F F F 18 360-379 110 110 110 176 176 176 F F F 19 380-399 107 107 107 173 173 173 F F F 20 400-419 110 110 110 176 176 176 F F F 21 420-439 110 110 110 176 176 176 F F F 22 440-459 106 106 106 170 170 170 F F F 23 460-479 109 109 109 175 175 175 F F F 24 480-499 109 109 109 175 175 175 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 110 110 110 176 176 176 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 110, Max flush: 176 Begin loop 56 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.061505702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.069998653Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070290655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070315124Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070323724Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070335343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Crucible cfcb8431-5584-41b5-9017-52a1b2b11c50 has session id: 3d83f9db-7252-4084-984a-2b9f181b012e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070631725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Upstairs opts: Upstairs UUID: cfcb8431-5584-41b5-9017-52a1b2b11c50, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070660724Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Crucible stats registered with UUID: cfcb8431-5584-41b5-9017-52a1b2b11c50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070673744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:54.070682924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070839209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070870058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070881318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.070992595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071002775Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"cfcb8431-5584-41b5-9017-52a1b2b11c50 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071048883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071058453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071068283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071078113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071087782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071100592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63170)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07115592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07118063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64348)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071325816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"ds_connection connected from Ok(127.0.0.1:52589)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071371804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071765833Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.071823102Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07187469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072092154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072111144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[0]R flush_numbers[0..12]: [152, 176, 176, 173, 169, 176, 175, 176, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072161762Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[0]R generation[0..12]: [94, 110, 110, 107, 106, 110, 109, 110, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072173592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072184691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[1]R flush_numbers[0..12]: [152, 176, 176, 173, 169, 176, 175, 176, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072195331Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[1]R generation[0..12]: [94, 110, 110, 107, 106, 110, 109, 110, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072205971Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072216931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[2]R flush_numbers[0..12]: [152, 176, 176, 173, 169, 176, 175, 176, 174, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07222754Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[2]R generation[0..12]: [94, 110, 110, 107, 106, 110, 109, 110, 108, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07223851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.07225188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Max found gen is 111","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072261389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Generation requested: 112 >= found:111","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072271049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Next flush: 177","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072280419Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072289819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"":"downstairs","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072299548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072308358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"0","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072317988Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"1","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072327617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"client":"2","":"downstairs","session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072337457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072346587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"cfcb8431-5584-41b5-9017-52a1b2b11c50 is now active with session: 3d83f9db-7252-4084-984a-2b9f181b012e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072356467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072365616Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072373236Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"session_id":"3d83f9db-7252-4084-984a-2b9f181b012e"} {"msg":"Activated sub_volume cfcb8431-5584-41b5-9017-52a1b2b11c50","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072382446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072390656Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.072754295Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} 01/30 Write block 385 len 40960 data: 11 9 9 11 12 10 9 12 12 11 02/30 Write block 495 len 8192 data: 10 9 03/30 Read block 277 len 36864 04/30 Write block 451 len 24576 data: 8 7 8 8 8 11 05/30 Write block 175 len 20480 data: 8 8 9 8 8 06/30 Read block 36 len 20480 07/30 Write block 441 len 12288 data: 8 9 9 08/30 Write block 85 len 24576 data: 8 7 9 8 7 7 09/30 Flush 10/30 Flush 11/30 Write block 7 len 12288 data: 10 10 7 12/30 Write block 96 len 28672 data: 9 7 5 6 7 7 7 13/30 Write block 431 len 8192 data: 11 7 14/30 Write block 445 len 36864 data: 15 13 12 11 10 7 9 8 9 15/30 Write block 76 len 28672 data: 7 7 7 8 6 8 8 16/30 Write block 480 len 4096 data: 5 17/30 Read block 554 len 12288 18/30 Write block 311 len 4096 data: 11 19/30 Read block 313 len 24576 20/30 Read block 506 len 40960 21/30 Write block 16 len 40960 data: 9 7 9 11 12 11 13 11 10 11 22/30 Write block 43 len 32768 data: 18 18 15 17 19 14 12 13 23/30 Read block 466 len 12288 24/30 Flush 25/30 Write block 563 len 12288 data: 8 9 8 26/30 Write block 381 len 36864 data: 9 8 10 11 12 10 10 12 13 27/30 Write block 264 len 16384 data: 7 10 8 9 28/30 Write block 253 len 32768 data: 12 12 9 9 7 7 8 6 29/30 Read block 526 len 32768 30/30 Read block 367 len 16384 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Write 2 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:54.081089291Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:54.081271816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1584,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 112 112 152 179 179 F F F <--- 1 020-039 110 112 112 176 179 179 F F F <--- 2 040-059 110 112 112 176 179 179 F F F <--- 3 060-079 107 112 112 173 179 179 F F F <--- 4 080-099 106 112 112 169 179 179 F F F <--- 5 100-119 110 112 112 176 179 179 F F F <--- 6 120-139 109 109 109 175 175 175 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 108 112 112 174 177 177 F F F <--- 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 110 110 110 176 176 176 F T T <--- 13 260-279 109 109 109 175 175 175 F T T <--- 14 280-299 109 109 109 175 175 175 F F F 15 300-319 110 112 112 176 179 179 F F F <--- 16 320-339 107 107 107 173 173 173 F F F 17 340-359 110 110 110 176 176 176 F F F 18 360-379 110 110 110 176 176 176 F F F 19 380-399 107 112 112 173 177 177 F T T <--- 20 400-419 110 110 110 176 176 176 F F F 21 420-439 110 112 112 176 179 179 F F F <--- 22 440-459 106 112 112 170 179 179 F F F <--- 23 460-479 109 109 109 175 175 175 F F F 24 480-499 109 112 112 175 179 179 F F F <--- 25 500-519 102 102 102 164 164 164 F F F 26 520-539 110 110 110 176 176 176 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 102 102 102 164 164 164 F T T <--- 29 580-599 107 107 107 173 173 173 F F F Max gen: 112, Max flush: 179 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 179 179 179 F F F 1 020-039 112 112 112 179 179 179 F F F 2 040-059 112 112 112 179 179 179 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 112 112 112 179 179 179 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 109 109 109 175 175 175 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 112 112 112 177 177 177 F F F 9 180-199 106 106 106 170 170 170 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 106 106 106 170 170 170 F F F 12 240-259 113 113 113 180 180 180 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 112 112 112 179 179 179 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 110 110 110 176 176 176 F F F 18 360-379 110 110 110 176 176 176 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 110 110 110 176 176 176 F F F 21 420-439 112 112 112 179 179 179 F F F 22 440-459 112 112 112 179 179 179 F F F 23 460-479 109 109 109 175 175 175 F F F 24 480-499 112 112 112 179 179 179 F F F 25 500-519 102 102 102 164 164 164 F F F 26 520-539 110 110 110 176 176 176 F F F 27 540-559 106 106 106 171 171 171 F F F 28 560-579 113 113 113 180 180 180 F F F 29 580-599 107 107 107 173 173 173 F F F Max gen: 113, Max flush: 180 Begin loop 57 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.350261601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.358578327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.358870089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.358895998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.358904848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.358916687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Crucible d0bb00fd-1a11-442f-89ca-86f1dfd58f81 has session id: 1b0d1f79-4ec5-496a-956c-f3522a2b2372","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359265078Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Upstairs opts: Upstairs UUID: d0bb00fd-1a11-442f-89ca-86f1dfd58f81, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359290207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Crucible stats registered with UUID: d0bb00fd-1a11-442f-89ca-86f1dfd58f81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359300387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:58.359309766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359471892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359488221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359501141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359613528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359629167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"d0bb00fd-1a11-442f-89ca-86f1dfd58f81 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359665506Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359675026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359685016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359694875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359704425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359714555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359724515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"ds_connection connected from Ok(127.0.0.1:47985)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359777933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"ds_connection connected from Ok(127.0.0.1:36977)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359793183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"ds_connection connected from Ok(127.0.0.1:47488)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.359803652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360531252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360547462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360633899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360869262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360900862Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[0]R flush_numbers[0..12]: [179, 179, 179, 179, 179, 179, 175, 176, 177, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360975949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[0]R generation[0..12]: [112, 112, 112, 112, 112, 112, 109, 110, 112, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.360987549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361009559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[1]R flush_numbers[0..12]: [179, 179, 179, 179, 179, 179, 175, 176, 177, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361020378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[1]R generation[0..12]: [112, 112, 112, 112, 112, 112, 109, 110, 112, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361031068Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361042258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[2]R flush_numbers[0..12]: [179, 179, 179, 179, 179, 179, 175, 176, 177, 170, 170, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361052847Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[2]R generation[0..12]: [112, 112, 112, 112, 112, 112, 109, 110, 112, 106, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361063467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361074267Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361083896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361093606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Next flush: 181","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361103076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361114076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"":"downstairs","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361123815Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361132635Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"0","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361142255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"1","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361151825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"client":"2","":"downstairs","session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361161724Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361170804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"d0bb00fd-1a11-442f-89ca-86f1dfd58f81 is now active with session: 1b0d1f79-4ec5-496a-956c-f3522a2b2372","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361180864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361189723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"session_id":"1b0d1f79-4ec5-496a-956c-f3522a2b2372"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361198863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Activated sub_volume d0bb00fd-1a11-442f-89ca-86f1dfd58f81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361206913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361215113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.361560913Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} 01/30 Write block 564 len 40960 data: 10 9 7 8 9 7 8 5 4 6 02/30 Read block 326 len 4096 03/30 Flush 04/30 Flush 05/30 Write block 592 len 28672 data: 8 6 7 5 5 4 4 06/30 Write block 239 len 32768 data: 12 12 10 10 9 9 8 12 07/30 Read block 241 len 16384 08/30 Write block 433 len 16384 data: 8 9 10 9 09/30 Write block 462 len 16384 data: 6 6 6 5 10/30 Flush 11/30 Write block 181 len 8192 data: 9 9 12/30 Write block 2 len 12288 data: 5 5 8 13/30 Write block 52 len 12288 data: 9 11 10 14/30 Flush 15/30 Write block 549 len 36864 data: 10 13 13 15 15 12 11 12 11 16/30 Flush 17/30 Write block 83 len 24576 data: 8 9 9 8 10 9 18/30 Read block 514 len 8192 19/30 Read block 327 len 28672 20/30 Flush 21/30 Write block 453 len 4096 data: 10 22/30 Read block 332 len 32768 23/30 Write block 560 len 16384 data: 10 10 10 9 24/30 Read block 147 len 16384 25/30 Read block 109 len 40960 26/30 Write block 500 len 16384 data: 6 7 6 8 27/30 Write block 358 len 36864 data: 7 9 8 9 6 7 8 13 11 28/30 Write block 133 len 24576 data: 3 4 4 5 7 8 29/30 Read block 235 len 36864 30/30 Read block 408 len 8192 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Read 1 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 8 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:12:58.370030705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:12:58.37020248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1592,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 114 114 112 184 184 179 F F F <--- 1 020-039 112 112 112 179 179 179 F F F 2 040-059 114 114 112 184 184 179 F F F <--- 3 060-079 112 112 112 179 179 179 F F F 4 080-099 114 114 112 186 186 179 F F F <--- 5 100-119 112 112 112 179 179 179 F F F 6 120-139 109 109 109 175 175 175 T T F <--- 7 140-159 110 110 110 176 176 176 F F F 8 160-179 112 112 112 177 177 177 F F F 9 180-199 114 114 106 184 184 170 F F F <--- 10 200-219 106 106 106 170 170 170 F F F 11 220-239 114 114 106 183 183 170 F F F <--- 12 240-259 114 114 113 183 183 180 F F F <--- 13 260-279 113 113 113 180 180 180 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 112 112 112 179 179 179 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 110 110 110 176 176 176 T T F <--- 18 360-379 110 110 110 176 176 176 T T F <--- 19 380-399 113 113 113 180 180 180 F F F 20 400-419 110 110 110 176 176 176 F F F 21 420-439 114 114 112 183 183 179 F F F <--- 22 440-459 112 112 112 179 179 179 T T F <--- 23 460-479 114 114 109 183 183 175 F F F <--- 24 480-499 112 112 112 179 179 179 F F F 25 500-519 102 102 102 164 164 164 T T F <--- 26 520-539 110 110 110 176 176 176 F F F 27 540-559 114 114 106 185 185 171 F F F <--- 28 560-579 114 114 113 181 181 180 T T F <--- 29 580-599 114 114 107 183 183 173 F F F <--- Max gen: 114, Max flush: 186 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 114 114 114 184 184 184 F F F 1 020-039 112 112 112 179 179 179 F F F 2 040-059 114 114 114 184 184 184 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 114 114 114 186 186 186 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 112 112 112 177 177 177 F F F 9 180-199 114 114 114 184 184 184 F F F 10 200-219 106 106 106 170 170 170 F F F 11 220-239 114 114 114 183 183 183 F F F 12 240-259 114 114 114 183 183 183 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 109 109 109 175 175 175 F F F 15 300-319 112 112 112 179 179 179 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 115 115 115 187 187 187 F F F 18 360-379 115 115 115 187 187 187 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 110 110 110 176 176 176 F F F 21 420-439 114 114 114 183 183 183 F F F 22 440-459 115 115 115 187 187 187 F F F 23 460-479 114 114 114 183 183 183 F F F 24 480-499 112 112 112 179 179 179 F F F 25 500-519 115 115 115 187 187 187 F F F 26 520-539 110 110 110 176 176 176 F F F 27 540-559 114 114 114 185 185 185 F F F 28 560-579 115 115 115 187 187 187 F F F 29 580-599 114 114 114 183 183 183 F F F Max gen: 115, Max flush: 187 Begin loop 58 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.640156048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648335338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648633449Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648657749Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648669658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648681408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Crucible dcc3a304-db15-4bf8-ba73-b049ac5bd76c has session id: 0c5a3cf3-31ef-4f23-9453-69c13e49d1ab","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.648993179Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Upstairs opts: Upstairs UUID: dcc3a304-db15-4bf8-ba73-b049ac5bd76c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649023198Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Crucible stats registered with UUID: dcc3a304-db15-4bf8-ba73-b049ac5bd76c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649034438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:02.649044048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649185124Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649201533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649211513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649345749Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649364079Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"dcc3a304-db15-4bf8-ba73-b049ac5bd76c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649388218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649397578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649407278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649419207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649428947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649438877Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649448796Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:52474)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649480965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:64438)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649495905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:59349)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.649624461Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650276523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650300152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.65039408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650653903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650682252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[0]R flush_numbers[0..12]: [184, 179, 184, 179, 186, 179, 187, 176, 177, 184, 170, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.65074987Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[0]R generation[0..12]: [114, 112, 114, 112, 114, 112, 115, 110, 112, 114, 106, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650761969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650773379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[1]R flush_numbers[0..12]: [184, 179, 184, 179, 186, 179, 187, 176, 177, 184, 170, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650786919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[1]R generation[0..12]: [114, 112, 114, 112, 114, 112, 115, 110, 112, 114, 106, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650797878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650809108Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[2]R flush_numbers[0..12]: [184, 179, 184, 179, 186, 179, 187, 176, 177, 184, 170, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650820088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[2]R generation[0..12]: [114, 112, 114, 112, 114, 112, 115, 110, 112, 114, 106, 114]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650830888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650842097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650851717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650861707Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Next flush: 188","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650871306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650880916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"":"downstairs","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650890876Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650899766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"0","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650909655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"1","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650919445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"client":"2","":"downstairs","session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650929365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650940094Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"dcc3a304-db15-4bf8-ba73-b049ac5bd76c is now active with session: 0c5a3cf3-31ef-4f23-9453-69c13e49d1ab","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650950354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650959264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"session_id":"0c5a3cf3-31ef-4f23-9453-69c13e49d1ab"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650968554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Activated sub_volume dcc3a304-db15-4bf8-ba73-b049ac5bd76c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650976683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.650984883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.651336513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} 01/30 Write block 306 len 40960 data: 11 11 11 11 11 12 13 11 10 9 02/30 Write block 519 len 12288 data: 13 14 12 03/30 Write block 405 len 16384 data: 7 7 7 8 04/30 Write block 341 len 20480 data: 10 8 9 8 7 05/30 Write block 53 len 16384 data: 12 11 9 11 06/30 Read block 403 len 20480 07/30 Read block 212 len 8192 08/30 Write block 518 len 16384 data: 12 14 15 13 09/30 Write block 179 len 36864 data: 9 8 10 10 10 12 11 12 13 10/30 Flush 11/30 Write block 83 len 28672 data: 9 10 10 9 11 10 8 12/30 Read block 111 len 20480 13/30 Read block 347 len 4096 14/30 Read block 366 len 20480 15/30 Flush 16/30 Write block 9 len 20480 data: 8 5 7 7 8 17/30 Read block 231 len 32768 18/30 Read block 358 len 16384 19/30 Read block 437 len 40960 20/30 Read block 426 len 16384 21/30 Read block 330 len 32768 22/30 Read block 164 len 8192 23/30 Read block 509 len 32768 24/30 Write block 298 len 20480 data: 8 7 9 9 7 25/30 Write block 215 len 28672 data: 8 9 7 8 8 9 11 26/30 Write block 465 len 4096 data: 6 27/30 Read block 290 len 16384 28/30 Read block 136 len 36864 29/30 Write block 216 len 12288 data: 10 8 9 30/30 Read block 550 len 12288 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 1 26 Done 30 5 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:02.659689199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:02.659867454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1600,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 114 114 114 184 184 184 T F T <--- 1 020-039 112 112 112 179 179 179 F F F 2 040-059 116 114 116 188 184 188 F T F <--- 3 060-079 112 112 112 179 179 179 F F F 4 080-099 116 114 116 189 186 189 F F F <--- 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 116 112 116 188 177 188 F F F <--- 9 180-199 116 114 116 188 184 188 F F F <--- 10 200-219 106 106 106 170 170 170 T F T <--- 11 220-239 114 114 114 183 183 183 T F T <--- 12 240-259 114 114 114 183 183 183 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 109 109 109 175 175 175 T F T <--- 15 300-319 116 112 116 188 179 188 T T T <--- 16 320-339 107 107 107 173 173 173 F F F 17 340-359 116 115 116 188 187 188 F T F <--- 18 360-379 115 115 115 187 187 187 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 116 110 116 188 176 188 F T F <--- 21 420-439 114 114 114 183 183 183 F F F 22 440-459 115 115 115 187 187 187 F F F 23 460-479 114 114 114 183 183 183 T F T <--- 24 480-499 112 112 112 179 179 179 F F F 25 500-519 116 115 116 188 187 188 F T F <--- 26 520-539 116 110 116 188 176 188 F T F <--- 27 540-559 114 114 114 185 185 185 F F F 28 560-579 115 115 115 187 187 187 F F F 29 580-599 114 114 114 183 183 183 F F F Max gen: 116, Max flush: 189 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 190 190 190 F F F 1 020-039 112 112 112 179 179 179 F F F 2 040-059 116 116 116 188 188 188 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 116 116 116 189 189 189 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 116 116 116 188 188 188 F F F 9 180-199 116 116 116 188 188 188 F F F 10 200-219 117 117 117 190 190 190 F F F 11 220-239 117 117 117 190 190 190 F F F 12 240-259 114 114 114 183 183 183 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 117 117 117 190 190 190 F F F 15 300-319 117 117 117 190 190 190 F F F 16 320-339 107 107 107 173 173 173 F F F 17 340-359 116 116 116 188 188 188 F F F 18 360-379 115 115 115 187 187 187 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 114 114 114 183 183 183 F F F 22 440-459 115 115 115 187 187 187 F F F 23 460-479 117 117 117 190 190 190 F F F 24 480-499 112 112 112 179 179 179 F F F 25 500-519 116 116 116 188 188 188 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 114 114 114 185 185 185 F F F 28 560-579 115 115 115 187 187 187 F F F 29 580-599 114 114 114 183 183 183 F F F Max gen: 117, Max flush: 190 Begin loop 59 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.921388097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:06.928578305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.928628564Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.935557369Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.935825282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.935850491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.935859011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93587055Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Crucible 2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40 has session id: 12a3f932-2a1d-4e38-8d78-e88c591db7ed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936190491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Upstairs opts: Upstairs UUID: 2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936214071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Crucible stats registered with UUID: 2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93622422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:06.93623321Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936396876Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936413495Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936423525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.9365828Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93659824Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936647628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936657218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936667048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936676898Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936686617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936696607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:61350)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936722956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936735566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:42376)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936753736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:35333)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.936810134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937370168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937394717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937520274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937643041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93765958Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[0]R flush_numbers[0..12]: [190, 179, 188, 179, 189, 179, 187, 176, 188, 188, 190, 190]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937795416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[0]R generation[0..12]: [117, 112, 116, 112, 116, 112, 115, 110, 116, 116, 117, 117]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937806676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937817846Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[1]R flush_numbers[0..12]: [190, 179, 188, 179, 189, 179, 187, 176, 188, 188, 190, 190]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937828565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[1]R generation[0..12]: [117, 112, 116, 112, 116, 112, 115, 110, 116, 116, 117, 117]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937839275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937850305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[2]R flush_numbers[0..12]: [190, 179, 188, 179, 189, 179, 187, 176, 188, 188, 190, 190]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937863814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[2]R generation[0..12]: [117, 112, 116, 112, 116, 112, 115, 110, 116, 116, 117, 117]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937874504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937885634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937895223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937904933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Next flush: 191","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937914243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937923773Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"":"downstairs","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937933682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937942562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"0","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937952362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"1","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937962262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"client":"2","":"downstairs","session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937972261Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937981481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40 is now active with session: 12a3f932-2a1d-4e38-8d78-e88c591db7ed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.937991581Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.9380003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"session_id":"12a3f932-2a1d-4e38-8d78-e88c591db7ed"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.9380121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "Activated sub_volume 2a0fcaa7-1ccf-447e-9f88-fc25dff0ec40","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93802584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.93803411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.938302382Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} 01/30 Write block 317 len 16384 data: 9 9 9 7 02/30 Write block 292 len 40960 data: 7 9 6 5 6 7 9 8 10 10 03/30 Write block 334 len 28672 data: 7 9 8 8 6 8 10 04/30 Read block 80 len 36864 05/30 Write block 563 len 20480 data: 10 11 10 8 9 06/30 Read block 294 len 28672 07/30 Write block 525 len 8192 data: 10 9 08/30 Write block 583 len 8192 data: 12 13 09/30 Read block 578 len 32768 10/30 Write block 184 len 12288 data: 13 12 13 11/30 Write block 208 len 4096 data: 7 12/30 Read block 124 len 20480 13/30 Read block 437 len 20480 14/30 Write block 23 len 40960 data: 12 11 12 11 11 10 10 11 11 10 15/30 Flush 16/30 Write block 158 len 24576 data: 5 6 8 10 11 10 17/30 Write block 236 len 16384 data: 8 9 11 13 18/30 Read block 127 len 36864 19/30 Write block 489 len 20480 data: 7 9 9 9 8 20/30 Write block 539 len 36864 data: 8 10 9 8 9 7 6 6 9 21/30 Read block 345 len 28672 22/30 Flush 23/30 Write block 468 len 24576 data: 5 5 8 8 10 10 24/30 Read block 296 len 20480 25/30 Read block 406 len 4096 26/30 Read block 539 len 8192 27/30 Read block 217 len 40960 28/30 Write block 564 len 8192 data: 12 11 29/30 Write block 11 len 12288 data: 8 8 9 30/30 Write block 475 len 36864 data: 10 11 7 5 5 6 6 10 9 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 5 28 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:06.947110474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:06.947289169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1608,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 190 190 190 F T T <--- 1 020-039 112 118 118 179 191 191 F F F <--- 2 040-059 116 116 116 188 188 188 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 116 116 116 189 189 189 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 110 118 118 176 192 192 F F F <--- 8 160-179 116 118 118 188 192 192 F F F <--- 9 180-199 116 118 118 188 191 191 F F F <--- 10 200-219 117 118 118 190 191 191 F F F <--- 11 220-239 117 118 118 190 192 192 F F F <--- 12 240-259 114 114 114 183 183 183 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 117 118 118 190 191 191 T F F <--- 15 300-319 117 118 118 190 191 191 T F F <--- 16 320-339 107 118 118 173 191 191 T F F <--- 17 340-359 116 118 118 188 191 191 T F F <--- 18 360-379 115 115 115 187 187 187 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 114 114 114 183 183 183 F F F 22 440-459 115 115 115 187 187 187 F F F 23 460-479 117 117 117 190 190 190 F T T <--- 24 480-499 112 118 118 179 192 192 F T T <--- 25 500-519 116 116 116 188 188 188 F F F 26 520-539 116 118 118 188 192 192 F F F <--- 27 540-559 114 118 118 185 192 192 F F F <--- 28 560-579 115 118 118 187 191 191 T T T <--- 29 580-599 114 118 118 183 191 191 F F F <--- Max gen: 118, Max flush: 192 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 119 119 119 193 193 193 F F F 1 020-039 118 118 118 191 191 191 F F F 2 040-059 116 116 116 188 188 188 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 116 116 116 189 189 189 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 118 118 118 191 191 191 F F F 10 200-219 118 118 118 191 191 191 F F F 11 220-239 118 118 118 192 192 192 F F F 12 240-259 114 114 114 183 183 183 F F F 13 260-279 113 113 113 180 180 180 F F F 14 280-299 118 118 118 191 191 191 F F F 15 300-319 118 118 118 191 191 191 F F F 16 320-339 118 118 118 191 191 191 F F F 17 340-359 118 118 118 191 191 191 F F F 18 360-379 115 115 115 187 187 187 F F F 19 380-399 113 113 113 180 180 180 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 114 114 114 183 183 183 F F F 22 440-459 115 115 115 187 187 187 F F F 23 460-479 119 119 119 193 193 193 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 116 116 116 188 188 188 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 119 119 119 193 193 193 F F F 29 580-599 118 118 118 191 191 191 F F F Max gen: 119, Max flush: 193 Begin loop 60 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.222605446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.230968651Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231277602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231306882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231315481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231327161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Crucible 977490b5-a705-4a60-8bae-44a8fc08cb9b has session id: fc3ff63a-be2f-4042-8dd9-890994bde7f8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231620713Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Upstairs opts: Upstairs UUID: 977490b5-a705-4a60-8bae-44a8fc08cb9b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231686741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Crucible stats registered with UUID: 977490b5-a705-4a60-8bae-44a8fc08cb9b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23171224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:11.2317226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231859826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231893725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.231904285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23207903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232095839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"977490b5-a705-4a60-8bae-44a8fc08cb9b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232113149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232121979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232131808Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232141608Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232151338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232161307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232171407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:64122)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232229196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:49764)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232254915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:37944)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232333393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232928716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.232963845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233061812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233218298Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233242947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[0]R flush_numbers[0..12]: [193, 191, 188, 179, 189, 179, 187, 192, 192, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233373813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[0]R generation[0..12]: [119, 118, 116, 112, 116, 112, 115, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233385203Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233396483Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[1]R flush_numbers[0..12]: [193, 191, 188, 179, 189, 179, 187, 192, 192, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233407092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[1]R generation[0..12]: [119, 118, 116, 112, 116, 112, 115, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233417912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233428892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[2]R flush_numbers[0..12]: [193, 191, 188, 179, 189, 179, 187, 192, 192, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233439692Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[2]R generation[0..12]: [119, 118, 116, 112, 116, 112, 115, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233450441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233461601Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233471121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23348262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Next flush: 194","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23349235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23350178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"":"downstairs","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.23351153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233520299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"0","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233530199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"1","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233539929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"client":"2","":"downstairs","session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233549788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233559038Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"977490b5-a705-4a60-8bae-44a8fc08cb9b is now active with session: fc3ff63a-be2f-4042-8dd9-890994bde7f8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233569118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233578028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"session_id":"fc3ff63a-be2f-4042-8dd9-890994bde7f8"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233587327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Activated sub_volume 977490b5-a705-4a60-8bae-44a8fc08cb9b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233595537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233603517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.233960467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} 01/30 Write block 445 len 16384 data: 16 14 13 12 02/30 Read block 295 len 28672 03/30 Read block 94 len 12288 04/30 Write block 324 len 20480 data: 10 9 13 12 11 05/30 Read block 318 len 36864 06/30 Write block 81 len 20480 data: 9 9 10 11 11 07/30 Write block 366 len 20480 data: 12 11 11 10 7 08/30 Read block 423 len 12288 09/30 Read block 551 len 24576 10/30 Write block 52 len 24576 data: 10 13 12 10 12 12 11/30 Write block 356 len 28672 data: 6 4 8 10 9 10 7 12/30 Write block 245 len 32768 data: 9 13 13 15 16 18 16 13 13/30 Read block 513 len 16384 14/30 Read block 413 len 4096 15/30 Write block 390 len 8192 data: 11 10 16/30 Write block 4 len 32768 data: 9 7 8 11 11 9 6 9 17/30 Write block 35 len 36864 data: 11 10 11 13 13 14 15 18 19 18/30 Read block 67 len 28672 19/30 Read block 258 len 16384 20/30 Read block 530 len 28672 21/30 Write block 420 len 12288 data: 10 8 10 22/30 Read block 228 len 40960 23/30 Write block 259 len 20480 data: 9 7 6 6 5 24/30 Flush 25/30 Write block 566 len 4096 data: 9 26/30 Flush 27/30 Write block 205 len 24576 data: 9 6 5 8 11 10 28/30 Write block 29 len 40960 data: 11 12 12 11 10 13 12 11 12 14 29/30 Write block 207 len 28672 data: 6 9 12 11 9 9 7 30/30 Write block 503 len 12288 data: 9 8 6 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Write 5 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 24 30 Done 27 27 6 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:11.24202662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:11.242208895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1616,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 119 194 194 193 F F F <--- 1 020-039 120 120 118 194 194 191 T T F <--- 2 040-059 120 120 116 194 194 188 F F F <--- 3 060-079 112 112 112 179 179 179 F F F 4 080-099 120 120 116 194 194 189 F F T <--- 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 118 118 118 191 191 191 F F F 10 200-219 118 118 118 191 191 191 T T F <--- 11 220-239 118 118 118 192 192 192 F F F 12 240-259 120 120 114 194 194 183 F F F <--- 13 260-279 120 120 113 194 194 180 F F F <--- 14 280-299 118 118 118 191 191 191 F F F 15 300-319 118 118 118 191 191 191 F F F 16 320-339 120 120 118 194 194 191 F F T <--- 17 340-359 120 120 118 194 194 191 F F F <--- 18 360-379 120 120 115 194 194 187 F F F <--- 19 380-399 120 120 113 194 194 180 F F F <--- 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 114 194 194 183 F F F <--- 22 440-459 120 120 115 194 194 187 F F T <--- 23 460-479 119 119 119 193 193 193 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 116 116 116 188 188 188 T T F <--- 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 119 195 195 193 F F F <--- 29 580-599 118 118 118 191 191 191 F F F Max gen: 120, Max flush: 195 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 121 121 121 196 196 196 F F F 2 040-059 120 120 120 194 194 194 F F F 3 060-079 112 112 112 179 179 179 F F F 4 080-099 120 120 120 194 194 194 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 115 115 115 187 187 187 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 118 118 118 191 191 191 F F F 10 200-219 121 121 121 196 196 196 F F F 11 220-239 118 118 118 192 192 192 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 118 118 118 191 191 191 F F F 15 300-319 118 118 118 191 191 191 F F F 16 320-339 120 120 120 194 194 194 F F F 17 340-359 120 120 120 194 194 194 F F F 18 360-379 120 120 120 194 194 194 F F F 19 380-399 120 120 120 194 194 194 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 120 194 194 194 F F F 22 440-459 120 120 120 194 194 194 F F F 23 460-479 119 119 119 193 193 193 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 121 121 121 196 196 196 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 F F F 29 580-599 118 118 118 191 191 191 F F F Max gen: 121, Max flush: 196 Begin loop 61 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.507365837Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.515645555Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.515977825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516012474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516021794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516033604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Crucible 01dd6588-82f2-4c4c-9743-d8b0868ac732 has session id: 55a02b9e-4b9f-48da-8dbf-d3a465f65d8f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516413263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Upstairs opts: Upstairs UUID: 01dd6588-82f2-4c4c-9743-d8b0868ac732, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516437012Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Crucible stats registered with UUID: 01dd6588-82f2-4c4c-9743-d8b0868ac732","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516447262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:15.516458932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516625637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516657646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516668606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516766513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516784523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"01dd6588-82f2-4c4c-9743-d8b0868ac732 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516808252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516819762Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516829611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516839431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516849161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516859221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.51686921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:52927)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516935858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:40612)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516947328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43633)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.516957728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.517639929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.517663208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.517768975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.517966119Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.517984489Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[0]R flush_numbers[0..12]: [194, 196, 194, 179, 194, 179, 187, 192, 192, 191, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518068777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[0]R generation[0..12]: [120, 121, 120, 112, 120, 112, 115, 118, 118, 118, 121, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518083136Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518094356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[1]R flush_numbers[0..12]: [194, 196, 194, 179, 194, 179, 187, 192, 192, 191, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518105086Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[1]R generation[0..12]: [120, 121, 120, 112, 120, 112, 115, 118, 118, 118, 121, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518115605Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518126485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[2]R flush_numbers[0..12]: [194, 196, 194, 179, 194, 179, 187, 192, 192, 191, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518136915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[2]R generation[0..12]: [120, 121, 120, 112, 120, 112, 115, 118, 118, 118, 121, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518147444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518158434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518167864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518177633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Next flush: 197","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518186973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518196183Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"":"downstairs","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518206043Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518214812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"0","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518225772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"1","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518235352Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"client":"2","":"downstairs","session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518245192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518254231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"01dd6588-82f2-4c4c-9743-d8b0868ac732 is now active with session: 55a02b9e-4b9f-48da-8dbf-d3a465f65d8f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518264131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518273041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"session_id":"55a02b9e-4b9f-48da-8dbf-d3a465f65d8f"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518281931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Activated sub_volume 01dd6588-82f2-4c4c-9743-d8b0868ac732","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.51828991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.51829829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.518683179Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} 01/30 Write block 366 len 32768 data: 13 12 12 11 8 10 8 7 02/30 Read block 50 len 24576 03/30 Write block 386 len 4096 data: 11 04/30 Write block 292 len 4096 data: 8 05/30 Read block 184 len 20480 06/30 Write block 478 len 4096 data: 6 07/30 Read block 520 len 4096 08/30 Read block 177 len 4096 09/30 Flush 10/30 Write block 518 len 8192 data: 13 15 11/30 Write block 195 len 36864 data: 11 10 10 12 14 11 11 10 11 12/30 Write block 61 len 4096 data: 8 13/30 Flush 14/30 Write block 120 len 16384 data: 14 15 13 13 15/30 Write block 324 len 16384 data: 11 10 14 13 16/30 Read block 372 len 8192 17/30 Read block 309 len 28672 18/30 Read block 467 len 24576 19/30 Read block 548 len 4096 20/30 Flush 21/30 Write block 68 len 20480 data: 7 6 7 5 4 22/30 Read block 332 len 32768 23/30 Read block 175 len 20480 24/30 Read block 123 len 28672 25/30 Read block 255 len 20480 26/30 Read block 34 len 12288 27/30 Write block 300 len 12288 data: 11 11 8 28/30 Read block 340 len 16384 29/30 Read block 577 len 36864 30/30 Write block 53 len 8192 data: 14 13 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Sent Done false 1001 Acked Read 6 Done Sent Done false 1002 Acked Write 1 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:15.527363705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:15.527595879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1624,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 121 121 121 196 196 196 F F F 2 040-059 120 120 120 194 194 194 T F T <--- 3 060-079 122 112 122 198 179 198 T F T <--- 4 080-099 120 120 120 194 194 194 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 122 115 122 199 187 199 F F F <--- 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 122 118 122 198 191 198 F F F <--- 10 200-219 122 121 122 198 196 198 F F F <--- 11 220-239 118 118 118 192 192 192 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 122 118 122 197 191 197 F F F <--- 15 300-319 118 118 118 191 191 191 T F T <--- 16 320-339 122 120 122 199 194 199 F F F <--- 17 340-359 120 120 120 194 194 194 F F F 18 360-379 122 120 122 197 194 197 F F F <--- 19 380-399 122 120 122 197 194 197 F F F <--- 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 120 194 194 194 F F F 22 440-459 120 120 120 194 194 194 F F F 23 460-479 122 119 122 197 193 197 F F F <--- 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 121 122 198 196 198 F F F <--- 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 F F F 29 580-599 118 118 118 191 191 191 F F F Max gen: 122, Max flush: 199 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 121 121 121 196 196 196 F F F 2 040-059 123 123 123 200 200 200 F F F 3 060-079 123 123 123 200 200 200 F F F 4 080-099 120 120 120 194 194 194 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 122 122 122 198 198 198 F F F 10 200-219 122 122 122 198 198 198 F F F 11 220-239 118 118 118 192 192 192 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 122 122 122 197 197 197 F F F 15 300-319 123 123 123 200 200 200 F F F 16 320-339 122 122 122 199 199 199 F F F 17 340-359 120 120 120 194 194 194 F F F 18 360-379 122 122 122 197 197 197 F F F 19 380-399 122 122 122 197 197 197 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 120 194 194 194 F F F 22 440-459 120 120 120 194 194 194 F F F 23 460-479 122 122 122 197 197 197 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 F F F 29 580-599 118 118 118 191 191 191 F F F Max gen: 123, Max flush: 200 Begin loop 62 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.787042602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795144394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795436406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795464175Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795472765Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795484354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Crucible e19e0b8f-a450-437f-b0d1-4672328243a9 has session id: 17fe477c-e171-4751-a40a-071c3dbf9660","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795823825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Upstairs opts: Upstairs UUID: e19e0b8f-a450-437f-b0d1-4672328243a9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795853544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Crucible stats registered with UUID: e19e0b8f-a450-437f-b0d1-4672328243a9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.795864944Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:19.795874243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796016999Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796046428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796057378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796208204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796222704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"e19e0b8f-a450-437f-b0d1-4672328243a9 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796240563Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796249443Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796259153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796268952Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796278632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796288552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796325151Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"ds_connection connected from Ok(127.0.0.1:36402)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"ds_connection connected from Ok(127.0.0.1:45091)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796394439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"ds_connection connected from Ok(127.0.0.1:54580)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.796493936Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.79707181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797090549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797167387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.79740582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.79742823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[0]R flush_numbers[0..12]: [194, 196, 200, 200, 194, 179, 199, 192, 192, 198, 198, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797440819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[0]R generation[0..12]: [120, 121, 123, 123, 120, 112, 122, 118, 118, 122, 122, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797452449Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797463619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[1]R flush_numbers[0..12]: [194, 196, 200, 200, 194, 179, 199, 192, 192, 198, 198, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797474338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[1]R generation[0..12]: [120, 121, 123, 123, 120, 112, 122, 118, 118, 122, 122, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797484948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797495998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[2]R flush_numbers[0..12]: [194, 196, 200, 200, 194, 179, 199, 192, 192, 198, 198, 192]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797508047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[2]R generation[0..12]: [120, 121, 123, 123, 120, 112, 122, 118, 118, 122, 122, 118]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797518547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797529677Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797539187Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797549146Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Next flush: 201","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797558576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797568176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"":"downstairs","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797578045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797586835Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"0","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797596405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"1","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797605985Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"client":"2","":"downstairs","session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797615804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797624984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"e19e0b8f-a450-437f-b0d1-4672328243a9 is now active with session: 17fe477c-e171-4751-a40a-071c3dbf9660","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797634994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797643894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"session_id":"17fe477c-e171-4751-a40a-071c3dbf9660"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797654673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Activated sub_volume e19e0b8f-a450-437f-b0d1-4672328243a9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797662583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.797670703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.798021343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} 01/30 Write block 228 len 4096 data: 8 02/30 Read block 430 len 36864 03/30 Write block 462 len 40960 data: 7 7 7 7 5 6 6 6 9 9 04/30 Read block 145 len 8192 05/30 Read block 295 len 24576 06/30 Flush 07/30 Write block 591 len 28672 data: 8 9 7 8 6 6 5 08/30 Read block 524 len 32768 09/30 Write block 584 len 20480 data: 14 11 10 9 10 10/30 Read block 58 len 28672 11/30 Write block 38 len 12288 data: 15 14 15 12/30 Read block 572 len 28672 13/30 Read block 194 len 4096 14/30 Read block 79 len 12288 15/30 Flush 16/30 Write block 106 len 4096 data: 6 17/30 Flush 18/30 Write block 348 len 24576 data: 9 8 8 9 11 10 19/30 Read block 473 len 28672 20/30 Flush 21/30 Write block 325 len 8192 data: 11 15 22/30 Write block 366 len 20480 data: 14 13 13 12 9 23/30 Read block 408 len 36864 24/30 Read block 2 len 24576 25/30 Flush 26/30 Write block 72 len 32768 data: 5 5 6 6 8 8 8 9 27/30 Read block 563 len 24576 28/30 Write block 380 len 28672 data: 8 10 9 11 12 13 12 29/30 Read block 556 len 28672 30/30 Read block 443 len 16384 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 9 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:19.80738931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:19.807568045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1632,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 124 121 124 202 196 202 F F F <--- 2 040-059 124 123 124 202 200 202 F F F <--- 3 060-079 123 123 123 200 200 200 T F T <--- 4 080-099 120 120 120 194 194 194 F F F 5 100-119 124 112 124 203 179 203 F F F <--- 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 122 122 122 198 198 198 F F F 10 200-219 122 122 122 198 198 198 F F F 11 220-239 124 118 124 201 192 201 F T F <--- 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 122 122 122 197 197 197 F F F 15 300-319 123 123 123 200 200 200 F F F 16 320-339 124 122 124 205 199 205 F F F <--- 17 340-359 124 120 124 204 194 204 F F F <--- 18 360-379 124 122 124 205 197 205 F F F <--- 19 380-399 122 122 122 197 197 197 T F T <--- 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 120 194 194 194 F F F 22 440-459 120 120 120 194 194 194 F F F 23 460-479 124 122 124 201 197 201 F F F <--- 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 F F F 29 580-599 124 118 124 202 191 202 F F F <--- Max gen: 124, Max flush: 205 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 124 124 124 202 202 202 F F F 2 040-059 124 124 124 202 202 202 F F F 3 060-079 125 125 125 206 206 206 F F F 4 080-099 120 120 120 194 194 194 F F F 5 100-119 124 124 124 203 203 203 F F F 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 118 118 118 192 192 192 F F F 9 180-199 122 122 122 198 198 198 F F F 10 200-219 122 122 122 198 198 198 F F F 11 220-239 124 124 124 201 201 201 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 122 122 122 197 197 197 F F F 15 300-319 123 123 123 200 200 200 F F F 16 320-339 124 124 124 205 205 205 F F F 17 340-359 124 124 124 204 204 204 F F F 18 360-379 124 124 124 205 205 205 F F F 19 380-399 125 125 125 206 206 206 F F F 20 400-419 116 116 116 188 188 188 F F F 21 420-439 120 120 120 194 194 194 F F F 22 440-459 120 120 120 194 194 194 F F F 23 460-479 124 124 124 201 201 201 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 F F F 29 580-599 124 124 124 202 202 202 F F F Max gen: 125, Max flush: 206 Begin loop 63 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.06297323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071084712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071379024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071403463Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071412003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071423513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Crucible c1ce023c-0f9d-441f-8cc4-2c21b0826129 has session id: 49c82c38-0ddf-421c-bb07-fe227620c56f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071746074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Upstairs opts: Upstairs UUID: c1ce023c-0f9d-441f-8cc4-2c21b0826129, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071777663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Crucible stats registered with UUID: c1ce023c-0f9d-441f-8cc4-2c21b0826129","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071788173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:24.071797262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071963728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.071999177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072010436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072118433Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072132103Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"c1ce023c-0f9d-441f-8cc4-2c21b0826129 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072166612Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072176072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072185741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072195421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072205051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072215021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.07222901Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57927)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072267509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54345)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072278259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54279)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.072297858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073107106Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073130235Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073149044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073425357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073455006Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[0]R flush_numbers[0..12]: [194, 202, 202, 206, 194, 203, 199, 192, 192, 198, 198, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073507494Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[0]R generation[0..12]: [120, 124, 124, 125, 120, 124, 122, 118, 118, 122, 122, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073519054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073530264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[1]R flush_numbers[0..12]: [194, 202, 202, 206, 194, 203, 199, 192, 192, 198, 198, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073540933Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[1]R generation[0..12]: [120, 124, 124, 125, 120, 124, 122, 118, 118, 122, 122, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073551583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073562613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[2]R flush_numbers[0..12]: [194, 202, 202, 206, 194, 203, 199, 192, 192, 198, 198, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073573232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[2]R generation[0..12]: [120, 124, 124, 125, 120, 124, 122, 118, 118, 122, 122, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073584062Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073595172Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073604562Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073614301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Next flush: 207","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073625961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073635501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"":"downstairs","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.07364521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.07365431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"0","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.07366413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"1","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.07367376Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"client":"2","":"downstairs","session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073683569Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073692599Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"c1ce023c-0f9d-441f-8cc4-2c21b0826129 is now active with session: 49c82c38-0ddf-421c-bb07-fe227620c56f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073702589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073711299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"session_id":"49c82c38-0ddf-421c-bb07-fe227620c56f"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073720518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Activated sub_volume c1ce023c-0f9d-441f-8cc4-2c21b0826129","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073728418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.073736528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.074108767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} 01/30 Flush 02/30 Flush 03/30 Write block 297 len 28672 data: 8 10 9 12 12 9 7 04/30 Read block 55 len 16384 05/30 Flush 06/30 Write block 191 len 16384 data: 9 8 10 10 07/30 Read block 365 len 28672 08/30 Write block 208 len 36864 data: 10 13 12 10 10 8 7 9 11 09/30 Read block 395 len 40960 10/30 Read block 196 len 8192 11/30 Write block 396 len 40960 data: 12 11 10 8 9 8 10 8 8 8 12/30 Read block 365 len 40960 13/30 Write block 435 len 24576 data: 11 10 8 7 6 6 14/30 Read block 200 len 16384 15/30 Write block 306 len 8192 data: 12 12 16/30 Write block 187 len 32768 data: 14 12 11 10 10 9 11 11 17/30 Write block 330 len 32768 data: 10 7 7 7 8 10 9 9 18/30 Write block 310 len 24576 data: 12 13 14 12 11 10 19/30 Flush 20/30 Write block 80 len 40960 data: 7 10 10 11 12 12 10 12 11 9 21/30 Write block 175 len 32768 data: 9 9 10 9 10 9 11 11 22/30 Read block 512 len 28672 23/30 Flush 24/30 Write block 563 len 8192 data: 11 13 25/30 Read block 110 len 12288 26/30 Read block 324 len 12288 27/30 Write block 571 len 32768 data: 6 5 7 6 7 8 10 12 28/30 Write block 163 len 4096 data: 11 29/30 Read block 267 len 28672 30/30 Write block 79 len 8192 data: 10 8 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 7 Done Done Done false 1003 Acked Read 4 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 27 27 1 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:24.083331778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:24.083538312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1640,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 124 124 124 202 202 202 F F F 2 040-059 124 124 124 202 202 202 F F F 3 060-079 125 125 125 206 206 206 T T F <--- 4 080-099 126 126 120 211 211 194 T T F <--- 5 100-119 124 124 124 203 203 203 F F F 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 126 126 118 211 211 192 T T F <--- 9 180-199 126 126 122 211 211 198 F F F <--- 10 200-219 126 126 122 210 210 198 F F F <--- 11 220-239 124 124 124 201 201 201 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 122 209 209 197 F F T <--- 15 300-319 126 126 123 210 210 200 F F T <--- 16 320-339 126 126 124 210 210 205 F F F <--- 17 340-359 124 124 124 204 204 204 F F F 18 360-379 124 124 124 205 205 205 F F F 19 380-399 126 126 125 210 210 206 F F F <--- 20 400-419 126 126 116 210 210 188 F F F <--- 21 420-439 126 126 120 210 210 194 F F F <--- 22 440-459 126 126 120 210 210 194 F F F <--- 23 460-479 124 124 124 201 201 201 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 120 120 120 195 195 195 T T F <--- 29 580-599 124 124 124 202 202 202 F F F Max gen: 126, Max flush: 211 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 124 124 124 202 202 202 F F F 2 040-059 124 124 124 202 202 202 F F F 3 060-079 127 127 127 212 212 212 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 124 124 124 203 203 203 F F F 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 127 127 127 212 212 212 F F F 9 180-199 126 126 126 211 211 211 F F F 10 200-219 126 126 126 210 210 210 F F F 11 220-239 124 124 124 201 201 201 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 126 126 126 210 210 210 F F F 17 340-359 124 124 124 204 204 204 F F F 18 360-379 124 124 124 205 205 205 F F F 19 380-399 126 126 126 210 210 210 F F F 20 400-419 126 126 126 210 210 210 F F F 21 420-439 126 126 126 210 210 210 F F F 22 440-459 126 126 126 210 210 210 F F F 23 460-479 124 124 124 201 201 201 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 124 124 124 202 202 202 F F F Max gen: 127, Max flush: 212 Begin loop 64 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.34424424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.3524336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.352725071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.352756111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35276479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35277671Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Crucible 94370175-ddea-44de-a8eb-0302bf0a4e46 has session id: a1ee454a-2721-4e97-ac51-896decb54d25","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353101801Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Upstairs opts: Upstairs UUID: 94370175-ddea-44de-a8eb-0302bf0a4e46, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35313066Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Crucible stats registered with UUID: 94370175-ddea-44de-a8eb-0302bf0a4e46","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35314195Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:28.353151169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353307805Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353337904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353348704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353456041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35347513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"94370175-ddea-44de-a8eb-0302bf0a4e46 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353505939Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353515469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353528349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353538329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353547918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353558128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353568058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"ds_connection connected from Ok(127.0.0.1:43799)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353633186Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"ds_connection connected from Ok(127.0.0.1:42337)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353651145Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"ds_connection connected from Ok(127.0.0.1:48819)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.353778162Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354347996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354365585Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354376595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354720845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354730465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[0]R flush_numbers[0..12]: [194, 202, 202, 212, 212, 203, 199, 192, 212, 211, 210, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354755694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[0]R generation[0..12]: [120, 124, 124, 127, 127, 124, 122, 118, 127, 126, 126, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354767244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354781054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[1]R flush_numbers[0..12]: [194, 202, 202, 212, 212, 203, 199, 192, 212, 211, 210, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354791663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[1]R generation[0..12]: [120, 124, 124, 127, 127, 124, 122, 118, 127, 126, 126, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354802113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354813123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[2]R flush_numbers[0..12]: [194, 202, 202, 212, 212, 203, 199, 192, 212, 211, 210, 201]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354823752Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[2]R generation[0..12]: [120, 124, 124, 127, 127, 124, 122, 118, 127, 126, 126, 124]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354834392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354845262Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354854652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354864541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Next flush: 213","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354873851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354883141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"":"downstairs","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35489325Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35490239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"0","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35491206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"1","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.35492157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"client":"2","":"downstairs","session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354932969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354942149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"94370175-ddea-44de-a8eb-0302bf0a4e46 is now active with session: a1ee454a-2721-4e97-ac51-896decb54d25","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354952169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354960819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"session_id":"a1ee454a-2721-4e97-ac51-896decb54d25"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354969948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Activated sub_volume 94370175-ddea-44de-a8eb-0302bf0a4e46","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354978018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.354986208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.355307309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} 01/30 Write block 366 len 28672 data: 15 14 14 13 10 11 9 02/30 Flush 03/30 Write block 350 len 32768 data: 9 10 12 11 8 7 7 5 04/30 Write block 24 len 20480 data: 12 13 12 12 11 05/30 Write block 366 len 28672 data: 16 15 15 14 11 12 10 06/30 Read block 112 len 32768 07/30 Write block 420 len 36864 data: 11 9 11 11 10 11 12 9 8 08/30 Flush 09/30 Write block 222 len 32768 data: 9 9 8 11 11 8 9 8 10/30 Write block 114 len 16384 data: 9 8 11 13 11/30 Write block 165 len 36864 data: 10 9 9 10 7 7 7 8 8 12/30 Read block 576 len 8192 13/30 Read block 99 len 24576 14/30 Read block 276 len 20480 15/30 Read block 255 len 12288 16/30 Read block 382 len 28672 17/30 Read block 580 len 16384 18/30 Write block 321 len 28672 data: 9 8 10 12 12 16 14 19/30 Read block 112 len 40960 20/30 Write block 14 len 8192 data: 10 10 21/30 Read block 171 len 32768 22/30 Write block 588 len 4096 data: 11 23/30 Read block 38 len 20480 24/30 Read block 501 len 20480 25/30 Read block 290 len 4096 26/30 Write block 14 len 28672 data: 11 11 10 8 10 12 13 27/30 Write block 510 len 12288 data: 5 7 8 28/30 Write block 207 len 8192 data: 7 11 29/30 Write block 398 len 40960 data: 11 9 10 9 11 9 9 9 8 8 30/30 Read block 493 len 12288 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 8 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 10 Done Sent Done false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 1 26 Done 28 3 27 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1001)) Some(JobId(1007)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:28.363910287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:28.364094962Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1648,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 T F T <--- 1 020-039 128 124 128 214 202 214 T T T <--- 2 040-059 124 124 124 202 202 202 F F F 3 060-079 127 127 127 212 212 212 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 124 124 124 203 203 203 T F T <--- 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 127 127 127 212 212 212 T F T <--- 9 180-199 126 126 126 211 211 211 F F F 10 200-219 126 126 126 210 210 210 T F T <--- 11 220-239 124 124 124 201 201 201 T F T <--- 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 126 126 126 210 210 210 T F T <--- 17 340-359 128 124 128 214 204 214 F T F <--- 18 360-379 128 128 128 214 213 214 F T F <--- 19 380-399 126 126 126 210 210 210 T F T <--- 20 400-419 126 126 126 210 210 210 T F T <--- 21 420-439 128 126 128 214 210 214 F F F <--- 22 440-459 126 126 126 210 210 210 F F F 23 460-479 124 124 124 201 201 201 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 122 122 122 198 198 198 T F T <--- 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 124 124 124 202 202 202 T F T <--- Max gen: 128, Max flush: 214 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 F F F 1 020-039 129 129 129 215 215 215 F F F 2 040-059 124 124 124 202 202 202 F F F 3 060-079 127 127 127 212 212 212 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 129 129 129 215 215 215 F F F 6 120-139 122 122 122 199 199 199 F F F 7 140-159 118 118 118 192 192 192 F F F 8 160-179 129 129 129 215 215 215 F F F 9 180-199 126 126 126 211 211 211 F F F 10 200-219 129 129 129 215 215 215 F F F 11 220-239 129 129 129 215 215 215 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 128 128 128 214 214 214 F F F 19 380-399 129 129 129 215 215 215 F F F 20 400-419 129 129 129 215 215 215 F F F 21 420-439 128 128 128 214 214 214 F F F 22 440-459 126 126 126 210 210 210 F F F 23 460-479 124 124 124 201 201 201 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 129 129 129 215 215 215 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 129 129 129 215 215 215 F F F Max gen: 129, Max flush: 215 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.628301111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63686903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637190311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63721702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63722592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.6372377Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Crucible ca154f6b-5cdf-4df8-a563-831041c708ec has session id: 5aa66d62-6fcc-4f71-8622-f00a82b0d30f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63759758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Upstairs opts: Upstairs UUID: ca154f6b-5cdf-4df8-a563-831041c708ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637641148Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Crucible stats registered with UUID: ca154f6b-5cdf-4df8-a563-831041c708ec","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637651728Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:32.637660718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637815763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637845013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637855902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637980049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.637993698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"ca154f6b-5cdf-4df8-a563-831041c708ec active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638028787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638038397Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638048217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638057867Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638067336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638077446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638087386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"ds_connection connected from Ok(127.0.0.1:44349)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638103055Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"ds_connection connected from Ok(127.0.0.1:44146)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638139174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50420)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638150304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638811155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638849974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.638862634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639141956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639188825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[0]R flush_numbers[0..12]: [215, 215, 202, 212, 212, 215, 199, 192, 215, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639235054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[0]R generation[0..12]: [129, 129, 124, 127, 127, 129, 122, 118, 129, 126, 129, 129]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639246263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639257383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[1]R flush_numbers[0..12]: [215, 215, 202, 212, 212, 215, 199, 192, 215, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639268283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[1]R generation[0..12]: [129, 129, 124, 127, 127, 129, 122, 118, 129, 126, 129, 129]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639278812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639289722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[2]R flush_numbers[0..12]: [215, 215, 202, 212, 212, 215, 199, 192, 215, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639302782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[2]R generation[0..12]: [129, 129, 124, 127, 127, 129, 122, 118, 129, 126, 129, 129]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639318411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639329391Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639338821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63934852Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Next flush: 216","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63935816Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63936753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"":"downstairs","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.63937712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639385799Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"0","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639395519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"1","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639405099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"client":"2","":"downstairs","session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639414878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639423928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"ca154f6b-5cdf-4df8-a563-831041c708ec is now active with session: 5aa66d62-6fcc-4f71-8622-f00a82b0d30f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639433938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639442758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"session_id":"5aa66d62-6fcc-4f71-8622-f00a82b0d30f"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639451817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Activated sub_volume ca154f6b-5cdf-4df8-a563-831041c708ec","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639461527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639469597Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.639874366Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} 01/30 Write block 144 len 4096 data: 10 02/30 Read block 503 len 36864 03/30 Write block 117 len 4096 data: 14 04/30 Read block 309 len 20480 05/30 Write block 462 len 8192 data: 8 8 06/30 Read block 516 len 40960 07/30 Read block 445 len 4096 08/30 Write block 231 len 20480 data: 7 5 7 9 10 09/30 Read block 250 len 40960 10/30 Read block 545 len 20480 11/30 Read block 136 len 20480 12/30 Read block 109 len 24576 13/30 Read block 93 len 24576 14/30 Write block 468 len 4096 data: 7 15/30 Flush 16/30 Write block 395 len 40960 data: 12 13 12 12 10 11 10 12 10 10 17/30 Write block 501 len 32768 data: 8 7 10 9 7 4 4 6 18/30 Read block 111 len 20480 19/30 Write block 151 len 12288 data: 15 14 13 20/30 Read block 455 len 24576 21/30 Read block 392 len 36864 22/30 Write block 52 len 36864 data: 11 15 14 11 13 13 11 12 9 23/30 Write block 205 len 36864 data: 10 7 8 12 14 13 11 11 9 24/30 Read block 410 len 12288 25/30 Write block 407 len 24576 data: 9 9 8 7 8 8 26/30 Write block 132 len 24576 data: 3 4 5 5 6 8 27/30 Read block 542 len 20480 28/30 Write block 217 len 40960 data: 9 10 9 10 12 10 10 9 12 12 29/30 Write block 392 len 20480 data: 13 13 12 13 14 30/30 Write block 162 len 8192 data: 12 12 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Write 2 Done Done Done false 1005 Acked Read 10 Done Done Done false 1006 Acked Read 1 Done Done Done false 1007 Acked Write 5 Done Done Done false 1008 Acked Read 10 Done Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 21 3 3 27 Done 9 27 27 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:32.648540142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:32.648747086Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1656,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 F F F 1 020-039 129 129 129 215 215 215 F F F 2 040-059 124 124 124 202 202 202 F T T <--- 3 060-079 127 127 127 212 212 212 F T T <--- 4 080-099 127 127 127 212 212 212 F F F 5 100-119 129 130 130 215 216 216 T F F <--- 6 120-139 122 122 122 199 199 199 F T T <--- 7 140-159 118 130 130 192 216 216 T T T <--- 8 160-179 129 129 129 215 215 215 F T T <--- 9 180-199 126 126 126 211 211 211 F F F 10 200-219 129 129 129 215 215 215 F T T <--- 11 220-239 129 130 130 215 216 216 T T T <--- 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 128 128 128 214 214 214 F F F 19 380-399 129 129 129 215 215 215 F T T <--- 20 400-419 129 129 129 215 215 215 F T T <--- 21 420-439 128 128 128 214 214 214 F F F 22 440-459 126 126 126 210 210 210 F F F 23 460-479 124 130 130 201 216 216 T F F <--- 24 480-499 119 119 119 193 193 193 F F F 25 500-519 129 129 129 215 215 215 F T T <--- 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 129 129 129 215 215 215 F F F Max gen: 130, Max flush: 216 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 F F F 1 020-039 129 129 129 215 215 215 F F F 2 040-059 131 131 131 217 217 217 F F F 3 060-079 131 131 131 217 217 217 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 130 130 130 216 216 216 F F F 6 120-139 131 131 131 217 217 217 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 131 131 131 217 217 217 F F F 9 180-199 126 126 126 211 211 211 F F F 10 200-219 131 131 131 217 217 217 F F F 11 220-239 131 131 131 217 217 217 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 128 128 128 214 214 214 F F F 19 380-399 131 131 131 217 217 217 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 128 128 128 214 214 214 F F F 22 440-459 126 126 126 210 210 210 F F F 23 460-479 130 130 130 216 216 216 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 131 131 131 217 217 217 F F F 26 520-539 118 118 118 192 192 192 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 129 129 129 215 215 215 F F F Max gen: 131, Max flush: 217 Begin loop 66 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.913712383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:36.92094285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.920992129Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.927791757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.92806343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928090549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928099279Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928110858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Crucible b8d7f03b-fa56-4c0c-8233-d41e263a5611 has session id: c02b2882-d26c-483b-a434-ed218c9bb849","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928429889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Upstairs opts: Upstairs UUID: b8d7f03b-fa56-4c0c-8233-d41e263a5611, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928458859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Crucible stats registered with UUID: b8d7f03b-fa56-4c0c-8233-d41e263a5611","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928470238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:36.928481768Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928628034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928644813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928654883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.92878141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928795089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"b8d7f03b-fa56-4c0c-8233-d41e263a5611 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928829948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928839448Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928849378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928859097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928868667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928897106Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928920366Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"ds_connection connected from Ok(127.0.0.1:37186)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928932855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"ds_connection connected from Ok(127.0.0.1:56529)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.928977974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"ds_connection connected from Ok(127.0.0.1:64197)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929073971Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929617386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929640345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929683504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929920708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.929939287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[0]R flush_numbers[0..12]: [215, 215, 217, 217, 212, 216, 217, 217, 217, 211, 217, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930036074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[0]R generation[0..12]: [129, 129, 131, 131, 127, 130, 131, 131, 131, 126, 131, 131]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930047584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930058764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[1]R flush_numbers[0..12]: [215, 215, 217, 217, 212, 216, 217, 217, 217, 211, 217, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930069703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[1]R generation[0..12]: [129, 129, 131, 131, 127, 130, 131, 131, 131, 126, 131, 131]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930080383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930091383Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[2]R flush_numbers[0..12]: [215, 215, 217, 217, 212, 216, 217, 217, 217, 211, 217, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930102012Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[2]R generation[0..12]: [129, 129, 131, 131, 127, 130, 131, 131, 131, 126, 131, 131]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930112542Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930123572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930135342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930145271Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Next flush: 218","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930154541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930163961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"":"downstairs","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.93017376Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.93018268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"0","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.93019237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"1","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.93020226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"client":"2","":"downstairs","session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930212109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930221429Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"b8d7f03b-fa56-4c0c-8233-d41e263a5611 is now active with session: c02b2882-d26c-483b-a434-ed218c9bb849","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930231609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930240469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"session_id":"c02b2882-d26c-483b-a434-ed218c9bb849"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930249618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Activated sub_volume b8d7f03b-fa56-4c0c-8233-d41e263a5611","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930257568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930265718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.930578049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} 01/30 Write block 527 len 8192 data: 10 8 02/30 Read block 542 len 4096 03/30 Read block 300 len 36864 04/30 Flush 05/30 Write block 458 len 4096 data: 10 06/30 Read block 12 len 16384 07/30 Read block 212 len 20480 08/30 Read block 182 len 28672 09/30 Write block 517 len 16384 data: 11 14 16 16 10/30 Write block 455 len 36864 data: 9 12 12 11 8 8 5 9 9 11/30 Flush 12/30 Flush 13/30 Write block 427 len 28672 data: 10 9 9 8 12 8 9 14/30 Write block 115 len 32768 data: 9 12 15 15 17 15 16 14 15/30 Write block 374 len 36864 data: 8 10 11 12 9 8 9 11 10 16/30 Read block 457 len 16384 17/30 Write block 217 len 28672 data: 10 11 10 11 13 11 11 18/30 Read block 48 len 8192 19/30 Write block 586 len 24576 data: 11 10 12 8 8 9 20/30 Read block 436 len 40960 21/30 Read block 467 len 4096 22/30 Read block 158 len 36864 23/30 Read block 439 len 32768 24/30 Flush 25/30 Flush 26/30 Write block 385 len 32768 data: 14 13 11 13 14 12 11 14 27/30 Write block 40 len 40960 data: 16 16 19 20 19 16 18 20 15 13 28/30 Read block 273 len 4096 29/30 Write block 23 len 36864 data: 13 13 14 13 13 12 12 13 13 30/30 Read block 161 len 12288 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:18 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1012 Acked Write 7 Done Done Done false 1013 Acked Write 8 Done Done Done false 1014 Acked Write 9 Done Done Done false 1015 Acked Read 4 Done Done Done false 1016 Acked Write 7 Done Done Done false 1017 Acked Read 2 Done Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 12 0 1 13 Done 6 18 17 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: 1011 1010 1009 1008 1007 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:36.940050603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:36.940230198Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1664,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 F F F 1 020-039 129 129 129 215 215 215 F T T <--- 2 040-059 131 131 131 217 217 217 F T T <--- 3 060-079 131 131 131 217 217 217 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 130 132 132 216 221 221 T F F <--- 6 120-139 131 132 132 217 221 221 T F F <--- 7 140-159 131 131 131 217 217 217 F F F 8 160-179 131 131 131 217 217 217 F F F 9 180-199 126 126 126 211 211 211 F F F 10 200-219 131 132 132 217 221 221 T F F <--- 11 220-239 131 132 132 217 221 221 T F F <--- 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 128 132 132 214 221 221 T F F <--- 19 380-399 131 132 132 217 221 221 T T T <--- 20 400-419 131 131 131 217 217 217 F F F 21 420-439 128 132 132 214 221 221 T F F <--- 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 132 132 132 219 219 219 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 129 132 132 215 221 221 F F F <--- Max gen: 132, Max flush: 221 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 F F F 1 020-039 133 133 133 222 222 222 F F F 2 040-059 133 133 133 222 222 222 F F F 3 060-079 131 131 131 217 217 217 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 132 132 132 221 221 221 F F F 6 120-139 132 132 132 221 221 221 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 131 131 131 217 217 217 F F F 9 180-199 126 126 126 211 211 211 F F F 10 200-219 132 132 132 221 221 221 F F F 11 220-239 132 132 132 221 221 221 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 126 126 126 209 209 209 F F F 15 300-319 126 126 126 210 210 210 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 132 132 132 221 221 221 F F F 19 380-399 133 133 133 222 222 222 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 132 132 132 221 221 221 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 132 132 132 219 219 219 F F F 27 540-559 118 118 118 192 192 192 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 132 132 132 221 221 221 F F F Max gen: 133, Max flush: 222 Begin loop 67 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.203289479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.211589426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.211875338Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.211901277Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.211912197Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.211923936Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Crucible b099172b-3aee-4a1e-9366-41ba3962a543 has session id: 0409fb5e-540e-4290-8df0-16f53e9c8a64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212263687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Upstairs opts: Upstairs UUID: b099172b-3aee-4a1e-9366-41ba3962a543, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212312375Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Crucible stats registered with UUID: b099172b-3aee-4a1e-9366-41ba3962a543","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212323305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:41.212332535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212484241Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21250031Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21251059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212623327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212636706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"b099172b-3aee-4a1e-9366-41ba3962a543 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212670105Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212679395Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212689095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212700505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212710184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212720214Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212730264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"ds_connection connected from Ok(127.0.0.1:49889)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212748313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"ds_connection connected from Ok(127.0.0.1:58882)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212761473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"ds_connection connected from Ok(127.0.0.1:50443)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.212779292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213408215Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213554431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21358083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213702576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213797334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[0]R flush_numbers[0..12]: [215, 222, 222, 217, 212, 221, 221, 217, 217, 211, 221, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213855472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[0]R generation[0..12]: [129, 133, 133, 131, 127, 132, 132, 131, 131, 126, 132, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213866772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213878051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[1]R flush_numbers[0..12]: [215, 222, 222, 217, 212, 221, 221, 217, 217, 211, 221, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213891451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[1]R generation[0..12]: [129, 133, 133, 131, 127, 132, 132, 131, 131, 126, 132, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213902441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21391355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[2]R flush_numbers[0..12]: [215, 222, 222, 217, 212, 221, 221, 217, 217, 211, 221, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21392451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[2]R generation[0..12]: [129, 133, 133, 131, 127, 132, 132, 131, 131, 126, 132, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21393519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.21394617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213955659Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213965529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Next flush: 223","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213974979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213984528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"":"downstairs","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.213994228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214002998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"0","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214012958Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"1","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214022617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"client":"2","":"downstairs","session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214034097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214043247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"b099172b-3aee-4a1e-9366-41ba3962a543 is now active with session: 0409fb5e-540e-4290-8df0-16f53e9c8a64","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214053347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214062406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214070026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"session_id":"0409fb5e-540e-4290-8df0-16f53e9c8a64"} {"msg":"Activated sub_volume b099172b-3aee-4a1e-9366-41ba3962a543","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214079326Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214087386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.214445825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} 01/30 Flush 02/30 Write block 66 len 24576 data: 9 8 8 7 8 6 03/30 Write block 101 len 36864 data: 8 8 5 7 6 7 8 10 11 04/30 Write block 200 len 16384 data: 12 12 11 12 05/30 Read block 11 len 16384 06/30 Write block 288 len 4096 data: 5 07/30 Read block 194 len 36864 08/30 Flush 09/30 Flush 10/30 Write block 312 len 32768 data: 15 13 12 11 8 10 10 10 11/30 Read block 161 len 4096 12/30 Write block 295 len 4096 data: 6 13/30 Read block 230 len 12288 14/30 Read block 525 len 4096 15/30 Read block 452 len 8192 16/30 Write block 193 len 36864 data: 12 12 12 11 11 13 15 13 13 17/30 Read block 503 len 32768 18/30 Write block 172 len 20480 data: 9 9 8 10 10 19/30 Write block 532 len 40960 data: 12 10 10 8 10 12 10 9 11 10 20/30 Read block 180 len 4096 21/30 Write block 12 len 4096 data: 9 22/30 Read block 591 len 24576 23/30 Read block 592 len 8192 24/30 Write block 392 len 28672 data: 15 14 13 14 15 13 13 25/30 Write block 130 len 16384 data: 5 3 4 5 26/30 Write block 120 len 4096 data: 16 27/30 Write block 194 len 16384 data: 13 13 12 12 28/30 Read block 544 len 12288 29/30 Write block 437 len 8192 data: 9 8 30/30 Read block 222 len 40960 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 29 0 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) Some(JobId(1000)) Some(JobId(1008)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:41.222046612Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:41.222255826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1672,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 129 129 129 215 215 215 T F T <--- 1 020-039 133 133 133 222 222 222 F F F 2 040-059 133 133 133 222 222 222 F F F 3 060-079 134 131 134 224 217 224 F F F <--- 4 080-099 127 127 127 212 212 212 F F F 5 100-119 134 132 134 224 221 224 F F F <--- 6 120-139 132 132 132 221 221 221 T F T <--- 7 140-159 131 131 131 217 217 217 F F F 8 160-179 131 131 131 217 217 217 T F T <--- 9 180-199 126 126 126 211 211 211 T F T <--- 10 200-219 134 132 134 224 221 224 T F T <--- 11 220-239 132 132 132 221 221 221 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 134 126 134 224 209 224 T F T <--- 15 300-319 126 126 126 210 210 210 T F T <--- 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 132 132 132 221 221 221 F F F 19 380-399 133 133 133 222 222 222 T F T <--- 20 400-419 131 131 131 217 217 217 F F F 21 420-439 132 132 132 221 221 221 T F T <--- 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 132 132 132 219 219 219 T F T <--- 27 540-559 118 118 118 192 192 192 T F T <--- 28 560-579 127 127 127 212 212 212 F F F 29 580-599 132 132 132 221 221 221 F F F Max gen: 134, Max flush: 224 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F F F 1 020-039 133 133 133 222 222 222 F F F 2 040-059 133 133 133 222 222 222 F F F 3 060-079 134 134 134 224 224 224 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 134 134 134 224 224 224 F F F 6 120-139 135 135 135 225 225 225 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 135 135 135 225 225 225 F F F 9 180-199 135 135 135 225 225 225 F F F 10 200-219 135 135 135 225 225 225 F F F 11 220-239 132 132 132 221 221 221 F F F 12 240-259 120 120 120 194 194 194 F F F 13 260-279 120 120 120 194 194 194 F F F 14 280-299 135 135 135 225 225 225 F F F 15 300-319 135 135 135 225 225 225 F F F 16 320-339 129 129 129 215 215 215 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 132 132 132 221 221 221 F F F 19 380-399 135 135 135 225 225 225 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 135 135 135 225 225 225 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 135 135 135 225 225 225 F F F 27 540-559 135 135 135 225 225 225 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 132 132 132 221 221 221 F F F Max gen: 135, Max flush: 225 Begin loop 68 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.483341623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.491468624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.491779496Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.491809915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.491818744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.491830474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Crucible 4a1760b7-a09a-4a7d-8e8e-729659cc4090 has session id: 3d78abbe-7428-4250-a0c3-04e8d051f932","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492129016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Upstairs opts: Upstairs UUID: 4a1760b7-a09a-4a7d-8e8e-729659cc4090, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492169045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Crucible stats registered with UUID: 4a1760b7-a09a-4a7d-8e8e-729659cc4090","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492182844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:45.492192004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.49234174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492357909Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492368039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492510805Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492537104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"4a1760b7-a09a-4a7d-8e8e-729659cc4090 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492546314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492555624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492566023Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492575993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492585683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492595863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492605882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"ds_connection connected from Ok(127.0.0.1:47433)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492646701Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"ds_connection connected from Ok(127.0.0.1:61723)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.492660431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"ds_connection connected from Ok(127.0.0.1:37094)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.49267087Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493314312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493340772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493424959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493640453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493650043Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[0]R flush_numbers[0..12]: [225, 222, 222, 224, 212, 224, 225, 217, 225, 225, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493725281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[0]R generation[0..12]: [135, 133, 133, 134, 127, 134, 135, 131, 135, 135, 135, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493736611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.49374756Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[1]R flush_numbers[0..12]: [225, 222, 222, 224, 212, 224, 225, 217, 225, 225, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.49375821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[1]R generation[0..12]: [135, 133, 133, 134, 127, 134, 135, 131, 135, 135, 135, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.49376869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493779539Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[2]R flush_numbers[0..12]: [225, 222, 222, 224, 212, 224, 225, 217, 225, 225, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493790199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[2]R generation[0..12]: [135, 133, 133, 134, 127, 134, 135, 131, 135, 135, 135, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493802969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493814098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493823588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493833268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Next flush: 226","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493842548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493851907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"":"downstairs","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493861997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493871027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"0","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493880866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"1","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493890676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"client":"2","":"downstairs","session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493900776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493909916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"4a1760b7-a09a-4a7d-8e8e-729659cc4090 is now active with session: 3d78abbe-7428-4250-a0c3-04e8d051f932","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493919855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493928465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"session_id":"3d78abbe-7428-4250-a0c3-04e8d051f932"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493937475Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Activated sub_volume 4a1760b7-a09a-4a7d-8e8e-729659cc4090","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493945465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.493955144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.494261306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} 01/30 Write block 168 len 24576 data: 11 8 8 8 10 10 02/30 Read block 0 len 12288 03/30 Read block 193 len 8192 04/30 Write block 102 len 24576 data: 9 6 8 7 8 9 05/30 Write block 421 len 16384 data: 10 12 12 11 06/30 Write block 226 len 36864 data: 13 9 10 9 8 8 6 8 10 07/30 Write block 230 len 40960 data: 9 9 7 9 11 11 9 10 12 14 08/30 Read block 115 len 40960 09/30 Write block 107 len 32768 data: 10 11 12 10 11 10 12 10 10/30 Flush 11/30 Write block 250 len 16384 data: 19 17 14 13 12/30 Write block 196 len 28672 data: 13 13 14 16 14 14 12 13/30 Write block 263 len 24576 data: 6 8 11 9 10 11 14/30 Flush 15/30 Write block 370 len 40960 data: 12 13 11 8 9 11 12 13 10 9 16/30 Read block 383 len 36864 17/30 Read block 353 len 12288 18/30 Read block 460 len 40960 19/30 Write block 37 len 8192 data: 13 16 20/30 Read block 287 len 16384 21/30 Read block 17 len 24576 22/30 Read block 415 len 12288 23/30 Read block 188 len 28672 24/30 Write block 65 len 16384 data: 9 10 9 9 25/30 Write block 324 len 36864 data: 13 13 17 15 12 13 11 8 8 26/30 Flush 27/30 Write block 528 len 16384 data: 9 8 9 11 28/30 Read block 531 len 40960 29/30 Read block 412 len 40960 30/30 Write block 585 len 20480 data: 12 12 11 13 9 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 16 18 Done 15 15 0 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) Some(JobId(1013)) Downstairs last five completed: 1013 1012 1011 1010 1009 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:45.503050399Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:45.503244473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1680,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F F F 1 020-039 136 136 133 228 228 222 F F F <--- 2 040-059 133 133 133 222 222 222 F F F 3 060-079 136 136 134 228 228 224 F F F <--- 4 080-099 127 127 127 212 212 212 F F F 5 100-119 136 136 136 226 226 226 F F F 6 120-139 135 135 135 225 225 225 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 136 136 136 226 226 226 F F F 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 136 136 136 226 226 226 F F F 12 240-259 136 136 136 227 227 227 F F F 13 260-279 136 136 136 227 227 227 F F F 14 280-299 135 135 135 225 225 225 F F F 15 300-319 135 135 135 225 225 225 F F F 16 320-339 136 136 129 228 228 215 F F F <--- 17 340-359 128 128 128 214 214 214 F F F 18 360-379 136 136 132 228 228 221 F F F <--- 19 380-399 135 135 135 225 225 225 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 136 136 136 226 226 226 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 135 135 135 225 225 225 T T F <--- 27 540-559 135 135 135 225 225 225 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 132 132 132 221 221 221 T T F <--- Max gen: 136, Max flush: 228 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 133 133 133 222 222 222 F F F 3 060-079 136 136 136 228 228 228 F F F 4 080-099 127 127 127 212 212 212 F F F 5 100-119 136 136 136 226 226 226 F F F 6 120-139 135 135 135 225 225 225 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 136 136 136 226 226 226 F F F 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 136 136 136 226 226 226 F F F 12 240-259 136 136 136 227 227 227 F F F 13 260-279 136 136 136 227 227 227 F F F 14 280-299 135 135 135 225 225 225 F F F 15 300-319 135 135 135 225 225 225 F F F 16 320-339 136 136 136 228 228 228 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 136 136 136 228 228 228 F F F 19 380-399 135 135 135 225 225 225 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 136 136 136 226 226 226 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 132 132 219 219 219 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 132 132 132 219 219 219 F F F 26 520-539 137 137 137 229 229 229 F F F 27 540-559 135 135 135 225 225 225 F F F 28 560-579 127 127 127 212 212 212 F F F 29 580-599 137 137 137 229 229 229 F F F Max gen: 137, Max flush: 229 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.745369753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:49.752557251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.7526057Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.759572344Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.759835336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.759862586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.759871315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.759882905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Crucible f40d6edc-c0d5-42fa-923d-14aeba8bd71c has session id: 7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760255065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Upstairs opts: Upstairs UUID: f40d6edc-c0d5-42fa-923d-14aeba8bd71c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760284424Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Crucible stats registered with UUID: f40d6edc-c0d5-42fa-923d-14aeba8bd71c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760295503Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:49.760304653Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760453229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760485248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760496348Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760603095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760626784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"f40d6edc-c0d5-42fa-923d-14aeba8bd71c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760653643Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760663533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760673633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760683403Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760692892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760702932Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760713062Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:33147)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.7607809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:44012)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760792269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:40248)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.760802769Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761532649Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761550798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761598557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.76185685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761868179Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[0]R flush_numbers[0..12]: [225, 228, 222, 228, 212, 226, 225, 217, 226, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761912478Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[0]R generation[0..12]: [135, 136, 133, 136, 127, 136, 135, 131, 136, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761923628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761934787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[1]R flush_numbers[0..12]: [225, 228, 222, 228, 212, 226, 225, 217, 226, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761945487Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[1]R generation[0..12]: [135, 136, 133, 136, 127, 136, 135, 131, 136, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761956047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761967076Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[2]R flush_numbers[0..12]: [225, 228, 222, 228, 212, 226, 225, 217, 226, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761977766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[2]R generation[0..12]: [135, 136, 133, 136, 127, 136, 135, 131, 136, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761988516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.761999476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762008845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762018535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Next flush: 230","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762027895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762037434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"":"downstairs","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762048884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762057654Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"0","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762067304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"1","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762077153Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"client":"2","":"downstairs","session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762087093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762096163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"f40d6edc-c0d5-42fa-923d-14aeba8bd71c is now active with session: 7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762106083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762114782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"session_id":"7c75b3d5-17f3-4c1a-a96e-3c41f2374ee5"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762123912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Activated sub_volume f40d6edc-c0d5-42fa-923d-14aeba8bd71c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762131832Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762139922Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.762463523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} 01/30 Write block 86 len 16384 data: 11 13 12 10 02/30 Read block 281 len 8192 03/30 Read block 327 len 40960 04/30 Read block 179 len 20480 05/30 Read block 552 len 40960 06/30 Read block 119 len 36864 07/30 Write block 473 len 28672 data: 11 11 11 12 8 7 6 08/30 Write block 245 len 12288 data: 10 14 14 09/30 Write block 496 len 32768 data: 10 10 9 6 7 9 8 11 10/30 Write block 77 len 24576 data: 9 9 11 9 11 11 11/30 Read block 220 len 16384 12/30 Write block 327 len 20480 data: 16 13 14 12 9 13/30 Flush 14/30 Write block 105 len 16384 data: 8 9 11 12 15/30 Read block 178 len 12288 16/30 Write block 510 len 12288 data: 6 8 9 17/30 Read block 174 len 20480 18/30 Read block 282 len 12288 19/30 Read block 355 len 8192 20/30 Write block 163 len 24576 data: 13 10 11 10 10 12 21/30 Write block 277 len 32768 data: 12 12 11 10 9 9 8 5 22/30 Read block 439 len 12288 23/30 Write block 305 len 20480 data: 8 13 13 12 12 24/30 Read block 166 len 28672 25/30 Read block 382 len 20480 26/30 Read block 570 len 20480 27/30 Write block 558 len 20480 data: 9 9 11 11 11 28/30 Write block 393 len 20480 data: 15 14 15 16 14 29/30 Write block 59 len 8192 data: 13 10 30/30 Read block 350 len 36864 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 1 27 Done 4 30 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:49.771417471Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:49.771605296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1688,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 133 133 133 222 222 222 F T T <--- 3 060-079 136 138 138 228 230 230 F T T <--- 4 080-099 127 138 138 212 230 230 T F F <--- 5 100-119 136 136 136 226 226 226 F T T <--- 6 120-139 135 135 135 225 225 225 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 136 136 136 226 226 226 F T T <--- 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 136 136 136 226 226 226 F F F 12 240-259 136 138 138 227 230 230 F F F <--- 13 260-279 136 136 136 227 227 227 F T T <--- 14 280-299 135 135 135 225 225 225 F T T <--- 15 300-319 135 135 135 225 225 225 F T T <--- 16 320-339 136 138 138 228 230 230 F F F <--- 17 340-359 128 128 128 214 214 214 F F F 18 360-379 136 136 136 228 228 228 F F F 19 380-399 135 135 135 225 225 225 F T T <--- 20 400-419 131 131 131 217 217 217 F F F 21 420-439 136 136 136 226 226 226 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 132 138 138 219 230 230 F F F <--- 24 480-499 119 138 138 193 230 230 F F F <--- 25 500-519 132 138 138 219 230 230 F T T <--- 26 520-539 137 137 137 229 229 229 F F F 27 540-559 135 135 135 225 225 225 F T T <--- 28 560-579 127 127 127 212 212 212 F T T <--- 29 580-599 137 137 137 229 229 229 F F F Max gen: 138, Max flush: 230 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 139 139 139 231 231 231 F F F 3 060-079 139 139 139 231 231 231 F F F 4 080-099 138 138 138 230 230 230 F F F 5 100-119 139 139 139 231 231 231 F F F 6 120-139 135 135 135 225 225 225 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 139 139 139 231 231 231 F F F 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 136 136 136 226 226 226 F F F 12 240-259 138 138 138 230 230 230 F F F 13 260-279 139 139 139 231 231 231 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 139 139 139 231 231 231 F F F 16 320-339 138 138 138 230 230 230 F F F 17 340-359 128 128 128 214 214 214 F F F 18 360-379 136 136 136 228 228 228 F F F 19 380-399 139 139 139 231 231 231 F F F 20 400-419 131 131 131 217 217 217 F F F 21 420-439 136 136 136 226 226 226 F F F 22 440-459 132 132 132 219 219 219 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 138 138 138 230 230 230 F F F 25 500-519 139 139 139 231 231 231 F F F 26 520-539 137 137 137 229 229 229 F F F 27 540-559 139 139 139 231 231 231 F F F 28 560-579 139 139 139 231 231 231 F F F 29 580-599 137 137 137 229 229 229 F F F Max gen: 139, Max flush: 231 Begin loop 70 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.043713112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:54.050849281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.0509001Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.05764448Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.057910843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.057937322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.057945922Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.057957621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Crucible c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1 has session id: bbe6bb95-f139-4941-920d-51dc374579bb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058293112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Upstairs opts: Upstairs UUID: c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058341191Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Crucible stats registered with UUID: c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.05835184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:54.05836106Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058513546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058534345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058544545Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058645142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058657742Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.05872193Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.05873154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058743309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058753099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058762759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058772739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058782748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:36333)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058822767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:63133)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058834017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:45097)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.058844336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059606095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059647284Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059709232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059921456Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059940766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[0]R flush_numbers[0..12]: [225, 228, 231, 231, 230, 231, 225, 217, 231, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.059991094Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[0]R generation[0..12]: [135, 136, 139, 139, 138, 139, 135, 131, 139, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060002504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060016314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[1]R flush_numbers[0..12]: [225, 228, 231, 231, 230, 231, 225, 217, 231, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060027053Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[1]R generation[0..12]: [135, 136, 139, 139, 138, 139, 135, 131, 139, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060037723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060048663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[2]R flush_numbers[0..12]: [225, 228, 231, 231, 230, 231, 225, 217, 231, 227, 227, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060059232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[2]R generation[0..12]: [135, 136, 139, 139, 138, 139, 135, 131, 139, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060069792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060080682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060089951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060099841Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Next flush: 232","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060109141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060118501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"":"downstairs","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.06012823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.06013729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"0","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.06014705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"1","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.06015823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"client":"2","":"downstairs","session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060168179Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060177269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1 is now active with session: bbe6bb95-f139-4941-920d-51dc374579bb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060187089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060196168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060203788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"session_id":"bbe6bb95-f139-4941-920d-51dc374579bb"} {"msg":"Activated sub_volume c6cee3c2-742b-4ab1-ad95-bc7f44f5c5d1","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060213048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060221148Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.060533729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} 01/30 Write block 575 len 12288 data: 8 9 11 02/30 Read block 183 len 36864 03/30 Write block 593 len 4096 data: 8 04/30 Read block 177 len 28672 05/30 Flush 06/30 Write block 404 len 20480 data: 11 10 9 10 10 07/30 Read block 466 len 16384 08/30 Write block 317 len 28672 data: 11 11 11 8 10 9 11 09/30 Write block 119 len 16384 data: 18 17 17 15 10/30 Read block 279 len 20480 11/30 Write block 571 len 24576 data: 7 6 8 7 9 10 12/30 Read block 381 len 8192 13/30 Write block 501 len 28672 data: 10 9 12 10 8 5 5 14/30 Read block 221 len 12288 15/30 Read block 22 len 28672 16/30 Write block 451 len 36864 data: 10 9 11 9 10 13 13 12 9 17/30 Read block 563 len 12288 18/30 Write block 222 len 20480 data: 12 12 10 13 14 19/30 Read block 306 len 16384 20/30 Flush 21/30 Write block 44 len 28672 data: 20 17 19 21 16 14 14 22/30 Read block 4 len 28672 23/30 Write block 10 len 28672 data: 7 10 10 10 12 12 11 24/30 Read block 464 len 40960 25/30 Write block 557 len 24576 data: 12 10 10 12 12 12 26/30 Read block 221 len 28672 27/30 Write block 379 len 32768 data: 10 10 12 11 12 13 15 14 28/30 Write block 108 len 28672 data: 13 13 11 12 11 13 11 29/30 Write block 345 len 16384 data: 8 8 9 10 30/30 Read block 343 len 16384 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:54.070234756Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:54.070422991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1696,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 225 225 225 F T T <--- 1 020-039 136 136 136 228 228 228 F F F 2 040-059 139 139 139 231 231 231 F T T <--- 3 060-079 139 139 139 231 231 231 F F F 4 080-099 138 138 138 230 230 230 F F F 5 100-119 139 140 140 231 233 233 F T T <--- 6 120-139 135 140 140 225 233 233 F F F <--- 7 140-159 131 131 131 217 217 217 F F F 8 160-179 139 139 139 231 231 231 F F F 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 136 140 140 226 233 233 F F F <--- 12 240-259 138 138 138 230 230 230 F F F 13 260-279 139 139 139 231 231 231 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 139 140 140 231 233 233 F F F <--- 16 320-339 138 140 140 230 233 233 F F F <--- 17 340-359 128 128 128 214 214 214 F T T <--- 18 360-379 136 136 136 228 228 228 F T T <--- 19 380-399 139 139 139 231 231 231 F T T <--- 20 400-419 131 140 140 217 233 233 F F F <--- 21 420-439 136 136 136 226 226 226 F F F 22 440-459 132 140 140 219 233 233 F F F <--- 23 460-479 138 138 138 230 230 230 F F F 24 480-499 138 138 138 230 230 230 F F F 25 500-519 139 140 140 231 233 233 F F F <--- 26 520-539 137 137 137 229 229 229 F F F 27 540-559 139 139 139 231 231 231 F T T <--- 28 560-579 139 140 140 231 233 233 F T T <--- 29 580-599 137 140 140 229 232 232 F F F <--- Max gen: 140, Max flush: 233 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 141 141 141 234 234 234 F F F 3 060-079 139 139 139 231 231 231 F F F 4 080-099 138 138 138 230 230 230 F F F 5 100-119 141 141 141 234 234 234 F F F 6 120-139 140 140 140 233 233 233 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 139 139 139 231 231 231 F F F 9 180-199 136 136 136 227 227 227 F F F 10 200-219 136 136 136 227 227 227 F F F 11 220-239 140 140 140 233 233 233 F F F 12 240-259 138 138 138 230 230 230 F F F 13 260-279 139 139 139 231 231 231 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 140 140 140 233 233 233 F F F 16 320-339 140 140 140 233 233 233 F F F 17 340-359 141 141 141 234 234 234 F F F 18 360-379 141 141 141 234 234 234 F F F 19 380-399 141 141 141 234 234 234 F F F 20 400-419 140 140 140 233 233 233 F F F 21 420-439 136 136 136 226 226 226 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 138 138 138 230 230 230 F F F 25 500-519 140 140 140 233 233 233 F F F 26 520-539 137 137 137 229 229 229 F F F 27 540-559 141 141 141 234 234 234 F F F 28 560-579 141 141 141 234 234 234 F F F 29 580-599 140 140 140 232 232 232 F F F Max gen: 141, Max flush: 234 Begin loop 71 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.345686099Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:58.352950455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.353007203Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.359817272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360084085Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360113434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360122193Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360136663Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Crucible f1f3285e-7a7c-4d2c-826d-0d9786cba156 has session id: 5680255d-bb8b-43c2-987c-51b57c638276","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360449904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Upstairs opts: Upstairs UUID: f1f3285e-7a7c-4d2c-826d-0d9786cba156, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360481423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Crucible stats registered with UUID: f1f3285e-7a7c-4d2c-826d-0d9786cba156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360492613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:58.360501943Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360653339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360671218Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360681348Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360790155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360804454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"f1f3285e-7a7c-4d2c-826d-0d9786cba156 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360827994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360837323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360847133Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360857013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360866813Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360878722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.360888812Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"ds_connection connected from Ok(127.0.0.1:44450)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.36096157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"ds_connection connected from Ok(127.0.0.1:50876)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361021158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"ds_connection connected from Ok(127.0.0.1:40580)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361079717Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361645851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361689559Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361709269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361962392Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.361998061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[0]R flush_numbers[0..12]: [234, 228, 234, 231, 230, 234, 233, 217, 231, 227, 227, 233]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362044139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[0]R generation[0..12]: [141, 136, 141, 139, 138, 141, 140, 131, 139, 136, 136, 140]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362056329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362067889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[1]R flush_numbers[0..12]: [234, 228, 234, 231, 230, 234, 233, 217, 231, 227, 227, 233]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362078888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[1]R generation[0..12]: [141, 136, 141, 139, 138, 141, 140, 131, 139, 136, 136, 140]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362091578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362102858Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[2]R flush_numbers[0..12]: [234, 228, 234, 231, 230, 234, 233, 217, 231, 227, 227, 233]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362113637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[2]R generation[0..12]: [141, 136, 141, 139, 138, 141, 140, 131, 139, 136, 136, 140]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362124087Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362135097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362144777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362154486Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362163926Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362173536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"":"downstairs","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362183416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362192245Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"0","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362202115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"1","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362211815Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"client":"2","":"downstairs","session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362221734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362230904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"f1f3285e-7a7c-4d2c-826d-0d9786cba156 is now active with session: 5680255d-bb8b-43c2-987c-51b57c638276","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362240794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362250964Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"session_id":"5680255d-bb8b-43c2-987c-51b57c638276"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362260253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Activated sub_volume f1f3285e-7a7c-4d2c-826d-0d9786cba156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362268223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362276283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.362578134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} 01/30 Write block 193 len 8192 data: 13 14 02/30 Write block 186 len 32768 data: 14 15 13 12 11 11 10 14 03/30 Write block 79 len 28672 data: 12 10 12 12 12 13 13 04/30 Write block 419 len 28672 data: 8 12 11 13 13 12 12 05/30 Write block 555 len 40960 data: 12 13 13 11 11 13 13 13 12 14 06/30 Read block 579 len 24576 07/30 Write block 212 len 40960 data: 12 10 8 10 12 11 12 11 12 14 08/30 Write block 484 len 4096 data: 8 09/30 Read block 141 len 12288 10/30 Read block 235 len 16384 11/30 Read block 223 len 4096 12/30 Write block 430 len 8192 data: 9 13 13/30 Flush 14/30 Write block 489 len 16384 data: 8 10 10 10 15/30 Read block 366 len 8192 16/30 Read block 121 len 40960 17/30 Write block 318 len 40960 data: 12 12 9 11 10 12 14 14 18 17 18/30 Write block 598 len 4096 data: 5 19/30 Read block 498 len 24576 20/30 Write block 587 len 4096 data: 12 21/30 Flush 22/30 Write block 492 len 16384 data: 11 9 9 11 23/30 Write block 543 len 8192 data: 10 8 24/30 Read block 586 len 24576 25/30 Read block 188 len 20480 26/30 Read block 324 len 8192 27/30 Write block 64 len 40960 data: 7 10 11 10 10 8 9 7 6 6 28/30 Write block 324 len 4096 data: 15 29/30 Read block 447 len 24576 30/30 Read block 187 len 36864 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:13:58.370406524Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:13:58.370589159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1704,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 141 141 141 234 234 234 F F F 3 060-079 139 142 142 231 235 235 F T T <--- 4 080-099 138 142 142 230 235 235 F F F <--- 5 100-119 141 141 141 234 234 234 F F F 6 120-139 140 140 140 233 233 233 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 139 139 139 231 231 231 F F F 9 180-199 136 142 142 227 235 235 T F F <--- 10 200-219 136 142 142 227 235 235 F F F <--- 11 220-239 140 142 142 233 235 235 F F F <--- 12 240-259 138 138 138 230 230 230 F F F 13 260-279 139 139 139 231 231 231 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 140 142 142 233 236 236 F F F <--- 16 320-339 140 142 142 233 236 236 F T T <--- 17 340-359 141 141 141 234 234 234 F F F 18 360-379 141 141 141 234 234 234 F F F 19 380-399 141 141 141 234 234 234 F F F 20 400-419 140 142 142 233 235 235 F F F <--- 21 420-439 136 142 142 226 235 235 F F F <--- 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 138 142 142 230 236 236 F T T <--- 25 500-519 140 140 140 233 233 233 F F F 26 520-539 137 137 137 229 229 229 F F F 27 540-559 141 142 142 234 235 235 F T T <--- 28 560-579 141 142 142 234 235 235 F F F <--- 29 580-599 140 142 142 232 236 236 F F F <--- Max gen: 142, Max flush: 236 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 141 141 141 234 234 234 F F F 3 060-079 143 143 143 237 237 237 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 141 141 141 234 234 234 F F F 6 120-139 140 140 140 233 233 233 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 139 139 139 231 231 231 F F F 9 180-199 142 142 142 235 235 235 F F F 10 200-219 142 142 142 235 235 235 F F F 11 220-239 142 142 142 235 235 235 F F F 12 240-259 138 138 138 230 230 230 F F F 13 260-279 139 139 139 231 231 231 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 142 142 142 236 236 236 F F F 16 320-339 143 143 143 237 237 237 F F F 17 340-359 141 141 141 234 234 234 F F F 18 360-379 141 141 141 234 234 234 F F F 19 380-399 141 141 141 234 234 234 F F F 20 400-419 142 142 142 235 235 235 F F F 21 420-439 142 142 142 235 235 235 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 143 143 143 237 237 237 F F F 25 500-519 140 140 140 233 233 233 F F F 26 520-539 137 137 137 229 229 229 F F F 27 540-559 143 143 143 237 237 237 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 142 142 142 236 236 236 F F F Max gen: 143, Max flush: 237 Begin loop 72 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.635229056Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.643490974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.643781836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.643805015Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.643813425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.643824974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Crucible 181f4202-ad3d-41dd-94b8-347ab456307f has session id: 25d14f6c-f191-47d9-88df-dcaacb56432b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644184734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Upstairs opts: Upstairs UUID: 181f4202-ad3d-41dd-94b8-347ab456307f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644209564Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Crucible stats registered with UUID: 181f4202-ad3d-41dd-94b8-347ab456307f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644219343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:02.644230703Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644391959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644422098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644433297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644532795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644549534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"181f4202-ad3d-41dd-94b8-347ab456307f active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644573333Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644582673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644592323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644601883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644611472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644621402Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644631332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"ds_connection connected from Ok(127.0.0.1:34326)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644643311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37304)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"ds_connection connected from Ok(127.0.0.1:46640)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.644798007Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645330182Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645362591Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645374421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645647223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645656963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[0]R flush_numbers[0..12]: [234, 228, 234, 237, 235, 234, 233, 217, 231, 235, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645722931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[0]R generation[0..12]: [141, 136, 141, 143, 142, 141, 140, 131, 139, 142, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645734731Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.64574603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[1]R flush_numbers[0..12]: [234, 228, 234, 237, 235, 234, 233, 217, 231, 235, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.64575711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[1]R generation[0..12]: [141, 136, 141, 143, 142, 141, 140, 131, 139, 142, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.64576785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.64577883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[2]R flush_numbers[0..12]: [234, 228, 234, 237, 235, 234, 233, 217, 231, 235, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645789539Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[2]R generation[0..12]: [141, 136, 141, 143, 142, 141, 140, 131, 139, 142, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645800269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645811529Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645823688Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645833838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Next flush: 238","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645843178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645852517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"":"downstairs","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645862837Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645871597Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"0","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645881657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"1","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645891686Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"client":"2","":"downstairs","session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645901836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645911186Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"181f4202-ad3d-41dd-94b8-347ab456307f is now active with session: 25d14f6c-f191-47d9-88df-dcaacb56432b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645921176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645929915Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"session_id":"25d14f6c-f191-47d9-88df-dcaacb56432b"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645939195Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Activated sub_volume 181f4202-ad3d-41dd-94b8-347ab456307f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645947275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.645955475Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.646329384Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} 01/30 Write block 508 len 40960 data: 7 6 7 9 10 9 10 9 11 12 02/30 Write block 345 len 32768 data: 9 9 10 11 9 10 11 13 03/30 Read block 9 len 32768 04/30 Flush 05/30 Write block 264 len 40960 data: 9 12 10 11 12 14 16 12 11 11 06/30 Read block 401 len 40960 07/30 Read block 231 len 32768 08/30 Read block 226 len 12288 09/30 Write block 249 len 32768 data: 17 20 18 15 14 13 10 10 10/30 Write block 549 len 16384 data: 11 14 14 16 11/30 Write block 42 len 8192 data: 20 21 12/30 Flush 13/30 Write block 177 len 28672 data: 11 10 11 10 12 12 11 14/30 Write block 426 len 12288 data: 13 11 10 15/30 Read block 488 len 8192 16/30 Write block 416 len 8192 data: 6 8 17/30 Read block 187 len 4096 18/30 Flush 19/30 Write block 173 len 4096 data: 11 20/30 Read block 127 len 36864 21/30 Flush 22/30 Write block 370 len 24576 data: 13 14 12 9 10 12 23/30 Read block 159 len 24576 24/30 Read block 357 len 16384 25/30 Read block 382 len 32768 26/30 Write block 117 len 24576 data: 16 16 19 18 18 16 27/30 Write block 532 len 24576 data: 13 11 11 9 11 13 28/30 Read block 312 len 8192 29/30 Read block 421 len 24576 30/30 Read block 37 len 32768 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 10 Done Done Done false 1005 Acked Read 10 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Write 8 Done Done Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 21 0 21 Done 26 5 26 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1003)) Some(JobId(1020)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:02.655500306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:02.655675211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1712,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 144 141 144 239 234 239 F F F <--- 3 060-079 143 143 143 237 237 237 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 141 141 141 234 234 234 T F T <--- 6 120-139 140 140 140 233 233 233 T F T <--- 7 140-159 131 131 131 217 217 217 F F F 8 160-179 144 139 144 241 231 241 F F F <--- 9 180-199 144 142 144 240 235 240 F F F <--- 10 200-219 142 142 142 235 235 235 F F F 11 220-239 142 142 142 235 235 235 F F F 12 240-259 144 138 144 239 230 239 F T F <--- 13 260-279 144 139 144 239 231 239 F T F <--- 14 280-299 139 139 139 231 231 231 F F F 15 300-319 142 142 142 236 236 236 F F F 16 320-339 143 143 143 237 237 237 F F F 17 340-359 144 144 144 238 238 238 F F F 18 360-379 141 141 141 234 234 234 T F T <--- 19 380-399 141 141 141 234 234 234 F F F 20 400-419 144 142 144 240 235 240 F F F <--- 21 420-439 144 142 144 240 235 240 F F F <--- 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 143 143 143 237 237 237 F F F 25 500-519 144 144 144 238 238 238 F F F 26 520-539 137 137 137 229 229 229 T F T <--- 27 540-559 144 143 144 239 237 239 F F F <--- 28 560-579 142 142 142 235 235 235 F F F 29 580-599 142 142 142 236 236 236 F F F Max gen: 144, Max flush: 241 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 144 144 144 239 239 239 F F F 3 060-079 143 143 143 237 237 237 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 145 145 145 242 242 242 F F F 6 120-139 145 145 145 242 242 242 F F F 7 140-159 131 131 131 217 217 217 F F F 8 160-179 144 144 144 241 241 241 F F F 9 180-199 144 144 144 240 240 240 F F F 10 200-219 142 142 142 235 235 235 F F F 11 220-239 142 142 142 235 235 235 F F F 12 240-259 144 144 144 239 239 239 F F F 13 260-279 144 144 144 239 239 239 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 142 142 142 236 236 236 F F F 16 320-339 143 143 143 237 237 237 F F F 17 340-359 144 144 144 238 238 238 F F F 18 360-379 145 145 145 242 242 242 F F F 19 380-399 141 141 141 234 234 234 F F F 20 400-419 144 144 144 240 240 240 F F F 21 420-439 144 144 144 240 240 240 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F F F 24 480-499 143 143 143 237 237 237 F F F 25 500-519 144 144 144 238 238 238 F F F 26 520-539 145 145 145 242 242 242 F F F 27 540-559 144 144 144 239 239 239 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 142 142 142 236 236 236 F F F Max gen: 145, Max flush: 242 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.912015981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92059033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.920887022Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.920913771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.920922521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92093447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Crucible 470f4a45-1c3c-401d-b2a8-e43e227e96a5 has session id: aa57e838-173c-4fe2-b14e-f9e5a51b7956","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921235172Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Upstairs opts: Upstairs UUID: 470f4a45-1c3c-401d-b2a8-e43e227e96a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921276711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Crucible stats registered with UUID: 470f4a45-1c3c-401d-b2a8-e43e227e96a5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92128756Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:06.9212968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921453336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921491595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921502684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921628131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92164565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"470f4a45-1c3c-401d-b2a8-e43e227e96a5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92166631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921675289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921685319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921695109Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921704919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921715028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921725148Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"ds_connection connected from Ok(127.0.0.1:56301)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921775797Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"ds_connection connected from Ok(127.0.0.1:49745)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921801286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"ds_connection connected from Ok(127.0.0.1:51579)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.921854534Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922420168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922445988Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922563504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92271592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92273362Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[0]R flush_numbers[0..12]: [234, 228, 239, 237, 235, 242, 242, 217, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922861046Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[0]R generation[0..12]: [141, 136, 144, 143, 142, 145, 145, 131, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922875096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922886525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[1]R flush_numbers[0..12]: [234, 228, 239, 237, 235, 242, 242, 217, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922897345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[1]R generation[0..12]: [141, 136, 144, 143, 142, 145, 145, 131, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922908065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922919144Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[2]R flush_numbers[0..12]: [234, 228, 239, 237, 235, 242, 242, 217, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922930064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[2]R generation[0..12]: [141, 136, 144, 143, 142, 145, 145, 131, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922940904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922951983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922961573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922971413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Next flush: 243","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922980873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.922990542Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"":"downstairs","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923000422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923009222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"0","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923020602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"1","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923030281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"client":"2","":"downstairs","session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923040171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.923049331Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"470f4a45-1c3c-401d-b2a8-e43e227e96a5 is now active with session: aa57e838-173c-4fe2-b14e-f9e5a51b7956","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92305954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92306879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.9230764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"session_id":"aa57e838-173c-4fe2-b14e-f9e5a51b7956"} {"msg":"Activated sub_volume 470f4a45-1c3c-401d-b2a8-e43e227e96a5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92308594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92309405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.92344152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} 01/30 Write block 352 len 8192 data: 14 12 02/30 Write block 68 len 28672 data: 11 9 10 8 7 7 7 03/30 Read block 275 len 12288 04/30 Read block 532 len 20480 05/30 Read block 156 len 20480 06/30 Read block 34 len 36864 07/30 Write block 378 len 8192 data: 11 11 08/30 Read block 149 len 40960 09/30 Read block 140 len 36864 10/30 Read block 178 len 24576 11/30 Write block 303 len 8192 data: 8 9 12/30 Read block 589 len 16384 13/30 Write block 110 len 20480 data: 12 13 12 14 12 14/30 Read block 161 len 24576 15/30 Write block 65 len 40960 data: 11 12 11 12 10 11 9 8 8 8 16/30 Write block 465 len 8192 data: 8 6 17/30 Write block 358 len 28672 data: 9 11 10 11 8 8 9 18/30 Read block 594 len 24576 19/30 Read block 246 len 24576 20/30 Write block 45 len 20480 data: 18 20 22 17 15 21/30 Write block 503 len 24576 data: 13 11 9 6 6 8 22/30 Write block 145 len 36864 data: 12 10 7 11 14 14 16 15 14 23/30 Read block 33 len 24576 24/30 Read block 537 len 28672 25/30 Read block 581 len 32768 26/30 Write block 581 len 24576 data: 10 13 13 15 13 13 27/30 Write block 346 len 24576 data: 10 11 12 10 11 12 28/30 Read block 19 len 36864 29/30 Read block 350 len 36864 30/30 Read block 415 len 20480 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 7 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:06.931643579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:06.931840514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1720,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 144 144 144 239 239 239 F T T <--- 3 060-079 143 143 143 237 237 237 F T T <--- 4 080-099 142 142 142 235 235 235 F F F 5 100-119 145 145 145 242 242 242 F T T <--- 6 120-139 145 145 145 242 242 242 F F F 7 140-159 131 131 131 217 217 217 F T T <--- 8 160-179 144 144 144 241 241 241 F F F 9 180-199 144 144 144 240 240 240 F F F 10 200-219 142 142 142 235 235 235 F F F 11 220-239 142 142 142 235 235 235 F F F 12 240-259 144 144 144 239 239 239 F F F 13 260-279 144 144 144 239 239 239 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 142 142 142 236 236 236 F T T <--- 16 320-339 143 143 143 237 237 237 F F F 17 340-359 144 144 144 238 238 238 T T T 18 360-379 145 145 145 242 242 242 F T T <--- 19 380-399 141 141 141 234 234 234 F F F 20 400-419 144 144 144 240 240 240 F F F 21 420-439 144 144 144 240 240 240 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 138 138 138 230 230 230 F T T <--- 24 480-499 143 143 143 237 237 237 F F F 25 500-519 144 144 144 238 238 238 F T T <--- 26 520-539 145 145 145 242 242 242 F F F 27 540-559 144 144 144 239 239 239 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 142 142 142 236 236 236 F T T <--- Max gen: 145, Max flush: 242 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now Files /tmp/test_repair/test_repair_verify.data and /tmp/test_repair/test_repair_verify.data.last differ 349d348 < 9, 352c351 < 9, --- > 12, 354a354 > 12, 361c361,362 < 8, --- > 9, > 11, 363d363 < 10, diff found after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 136 136 136 228 228 228 F F F 2 040-059 146 146 146 243 243 243 F F F 3 060-079 146 146 146 243 243 243 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 146 146 146 243 243 243 F F F 6 120-139 145 145 145 242 242 242 F F F 7 140-159 146 146 146 243 243 243 F F F 8 160-179 144 144 144 241 241 241 F F F 9 180-199 144 144 144 240 240 240 F F F 10 200-219 142 142 142 235 235 235 F F F 11 220-239 142 142 142 235 235 235 F F F 12 240-259 144 144 144 239 239 239 F F F 13 260-279 144 144 144 239 239 239 F F F 14 280-299 139 139 139 231 231 231 F F F 15 300-319 146 146 146 243 243 243 F F F 16 320-339 143 143 143 237 237 237 F F F 17 340-359 146 146 146 243 243 243 F F F 18 360-379 146 146 146 243 243 243 F F F 19 380-399 141 141 141 234 234 234 F F F 20 400-419 144 144 144 240 240 240 F F F 21 420-439 144 144 144 240 240 240 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 143 143 143 237 237 237 F F F 25 500-519 146 146 146 243 243 243 F F F 26 520-539 145 145 145 242 242 242 F F F 27 540-559 144 144 144 239 239 239 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 146 146 146 243 243 243 F F F Max gen: 146, Max flush: 243 Begin loop 74 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.196450451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.204739238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.20504902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205074129Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205082839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205097238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Crucible bb887dc5-60f7-48fa-ae54-2d98b772ddfb has session id: b2d3ec90-3627-45d5-a03c-c17eead3b30c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205456258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Upstairs opts: Upstairs UUID: bb887dc5-60f7-48fa-ae54-2d98b772ddfb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205514197Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Crucible stats registered with UUID: bb887dc5-60f7-48fa-ae54-2d98b772ddfb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205539436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:11.205549706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205695121Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205725771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.2057367Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205855467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205874326Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"bb887dc5-60f7-48fa-ae54-2d98b772ddfb active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205907435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205917165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205927025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205936725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205946544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.205959104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206002173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:39418)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206027172Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:61964)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206038702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46485)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206084311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206735462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206756852Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.206769411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207080753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207128681Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[0]R flush_numbers[0..12]: [234, 228, 243, 243, 235, 243, 242, 243, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.20718079Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[0]R generation[0..12]: [141, 136, 146, 146, 142, 146, 145, 146, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207192299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207203389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[1]R flush_numbers[0..12]: [234, 228, 243, 243, 235, 243, 242, 243, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207214239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[1]R generation[0..12]: [141, 136, 146, 146, 142, 146, 145, 146, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207227358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207238308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[2]R flush_numbers[0..12]: [234, 228, 243, 243, 235, 243, 242, 243, 241, 240, 235, 235]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207248848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[2]R generation[0..12]: [141, 136, 146, 146, 142, 146, 145, 146, 144, 144, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207259347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207270297Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Max found gen is 147","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207279637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Generation requested: 148 >= found:147","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207289117Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Next flush: 244","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207298536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207307926Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"":"downstairs","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207317836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207326636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"0","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207336485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"1","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207346145Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"client":"2","":"downstairs","session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207355865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207364985Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"bb887dc5-60f7-48fa-ae54-2d98b772ddfb is now active with session: b2d3ec90-3627-45d5-a03c-c17eead3b30c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207376304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207385414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207392954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"session_id":"b2d3ec90-3627-45d5-a03c-c17eead3b30c"} {"msg":"Activated sub_volume bb887dc5-60f7-48fa-ae54-2d98b772ddfb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207401993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207410083Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.207771513Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} 01/30 Write block 164 len 4096 data: 11 02/30 Flush 03/30 Write block 261 len 16384 data: 7 7 7 10 04/30 Write block 549 len 20480 data: 12 15 15 17 16 05/30 Read block 21 len 16384 06/30 Flush 07/30 Write block 393 len 40960 data: 16 15 16 17 15 14 11 12 11 13 08/30 Write block 42 len 20480 data: 21 22 21 19 21 09/30 Flush 10/30 Write block 352 len 28672 data: 15 13 9 8 8 6 9 11/30 Read block 17 len 24576 12/30 Write block 30 len 40960 data: 14 14 12 11 14 13 12 14 17 15 13/30 Read block 33 len 36864 14/30 Read block 333 len 40960 15/30 Write block 588 len 28672 data: 14 10 9 10 10 9 9 16/30 Read block 73 len 4096 17/30 Flush 18/30 Flush 19/30 Write block 295 len 32768 data: 7 7 9 11 10 13 13 10 20/30 Write block 482 len 40960 data: 11 10 9 7 6 7 7 9 11 11 21/30 Write block 292 len 20480 data: 9 10 7 8 8 22/30 Read block 83 len 32768 23/30 Read block 18 len 4096 24/30 Flush 25/30 Write block 212 len 24576 data: 13 11 9 11 13 12 26/30 Read block 314 len 16384 27/30 Write block 239 len 20480 data: 15 13 11 11 10 28/30 Write block 32 len 16384 data: 13 12 15 14 29/30 Write block 200 len 36864 data: 15 15 13 13 10 11 8 9 13 30/30 Write block 338 len 16384 data: 7 9 11 11 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1018 Acked Write 8 Done Done Done false 1019 Acked Write 10 Done Done Done false 1020 Acked Write 5 Done Done Done false 1021 Acked Read 8 Done Done Done false 1022 Acked Read 1 Done Done Done false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 7 11 Done 10 10 5 25 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1017)) Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:11.21678977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:11.216954865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1731,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 148 148 148 247 247 247 T T F <--- 2 040-059 148 148 148 246 246 246 F F F 3 060-079 146 146 146 243 243 243 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 146 146 146 243 243 243 F F F 6 120-139 145 145 145 242 242 242 F F F 7 140-159 146 146 146 243 243 243 F F F 8 160-179 148 148 148 244 244 244 F F F 9 180-199 144 144 144 240 240 240 F F F 10 200-219 142 142 142 235 235 235 T T F <--- 11 220-239 142 142 142 235 235 235 T T F <--- 12 240-259 144 144 144 239 239 239 T T F <--- 13 260-279 148 148 148 245 245 245 F F F 14 280-299 148 148 139 249 249 231 F F T <--- 15 300-319 148 148 146 249 249 243 F F T <--- 16 320-339 143 143 143 237 237 237 T T F <--- 17 340-359 148 148 148 247 247 247 T T F <--- 18 360-379 146 146 146 243 243 243 F F F 19 380-399 148 148 148 246 246 246 F F F 20 400-419 148 148 148 246 246 246 F F F 21 420-439 144 144 144 240 240 240 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 143 249 249 237 F F T <--- 25 500-519 146 146 146 243 243 243 F F F 26 520-539 145 145 145 242 242 242 F F F 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 148 148 148 247 247 247 F F F Max gen: 148, Max flush: 249 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 234 234 234 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 148 148 148 246 246 246 F F F 3 060-079 146 146 146 243 243 243 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 146 146 146 243 243 243 F F F 6 120-139 145 145 145 242 242 242 F F F 7 140-159 146 146 146 243 243 243 F F F 8 160-179 148 148 148 244 244 244 F F F 9 180-199 144 144 144 240 240 240 F F F 10 200-219 149 149 149 250 250 250 F F F 11 220-239 149 149 149 250 250 250 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 148 148 148 245 245 245 F F F 14 280-299 148 148 148 249 249 249 F F F 15 300-319 148 148 148 249 249 249 F F F 16 320-339 149 149 149 250 250 250 F F F 17 340-359 149 149 149 250 250 250 F F F 18 360-379 146 146 146 243 243 243 F F F 19 380-399 148 148 148 246 246 246 F F F 20 400-419 148 148 148 246 246 246 F F F 21 420-439 144 144 144 240 240 240 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 146 146 146 243 243 243 F F F 26 520-539 145 145 145 242 242 242 F F F 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 148 148 148 247 247 247 F F F Max gen: 149, Max flush: 250 Begin loop 75 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.470134584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.478529458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.478836159Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.478871108Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.478880488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.478892298Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Crucible 556b43dd-7a23-40e3-a080-182623ba50a4 has session id: b3d813e5-b774-47f9-9f20-f0c123bb5f24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479238408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Upstairs opts: Upstairs UUID: 556b43dd-7a23-40e3-a080-182623ba50a4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479276967Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Crucible stats registered with UUID: 556b43dd-7a23-40e3-a080-182623ba50a4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479288587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:15.479298116Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479467232Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479484661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479497301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479626487Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479638337Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"556b43dd-7a23-40e3-a080-182623ba50a4 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479670606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479680166Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479689895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479699525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479709365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479719515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479729454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"ds_connection connected from Ok(127.0.0.1:45013)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479768393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"ds_connection connected from Ok(127.0.0.1:54968)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479815032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"ds_connection connected from Ok(127.0.0.1:38510)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.479834311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480494963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480509212Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480631169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480858073Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480867802Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[0]R flush_numbers[0..12]: [234, 250, 246, 243, 235, 243, 242, 243, 244, 240, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.48093411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[0]R generation[0..12]: [141, 149, 148, 146, 142, 146, 145, 146, 148, 144, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.48094552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.48095659Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[1]R flush_numbers[0..12]: [234, 250, 246, 243, 235, 243, 242, 243, 244, 240, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480967319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[1]R generation[0..12]: [141, 149, 148, 146, 142, 146, 145, 146, 148, 144, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480977979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480989049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[2]R flush_numbers[0..12]: [234, 250, 246, 243, 235, 243, 242, 243, 244, 240, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.480999759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[2]R generation[0..12]: [141, 149, 148, 146, 142, 146, 145, 146, 148, 144, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481010418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481021628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481031238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481041157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Next flush: 251","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481050547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481062057Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"":"downstairs","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481071907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481080796Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"0","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481090316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"1","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481099976Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"client":"2","":"downstairs","session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481109855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481119015Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"556b43dd-7a23-40e3-a080-182623ba50a4 is now active with session: b3d813e5-b774-47f9-9f20-f0c123bb5f24","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481129085Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481137965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"session_id":"b3d813e5-b774-47f9-9f20-f0c123bb5f24"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481147334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Activated sub_volume 556b43dd-7a23-40e3-a080-182623ba50a4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481155554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481163734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.481555533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} 01/30 Write block 503 len 28672 data: 14 12 10 7 7 9 7 02/30 Write block 376 len 28672 data: 13 14 12 12 11 13 12 03/30 Write block 10 len 24576 data: 8 11 11 11 13 13 04/30 Write block 67 len 16384 data: 12 13 11 12 05/30 Read block 513 len 16384 06/30 Flush 07/30 Write block 189 len 40960 data: 13 12 12 11 15 15 14 14 14 15 08/30 Write block 111 len 12288 data: 14 13 15 09/30 Write block 507 len 12288 data: 8 10 8 10/30 Write block 417 len 12288 data: 9 8 9 11/30 Read block 295 len 8192 12/30 Write block 358 len 32768 data: 10 11 11 12 9 9 10 14 13/30 Read block 516 len 36864 14/30 Write block 352 len 36864 data: 16 14 10 9 9 7 11 12 12 15/30 Read block 354 len 28672 16/30 Write block 414 len 32768 data: 7 7 7 10 9 10 13 12 17/30 Flush 18/30 Write block 6 len 20480 data: 9 12 12 10 9 19/30 Write block 329 len 24576 data: 15 13 10 9 8 9 20/30 Read block 98 len 4096 21/30 Read block 541 len 36864 22/30 Write block 45 len 28672 data: 20 22 23 18 16 15 12 23/30 Read block 89 len 28672 24/30 Read block 302 len 32768 25/30 Write block 141 len 24576 data: 9 8 8 11 13 11 26/30 Write block 48 len 28672 data: 19 17 16 13 12 16 15 27/30 Write block 262 len 40960 data: 8 8 11 13 11 12 13 15 17 13 28/30 Read block 569 len 32768 29/30 Read block 200 len 28672 30/30 Write block 323 len 24576 data: 13 16 15 19 18 14 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Sent false 1001 Acked Write 7 Done Done Sent false 1002 Acked Write 6 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:15.49053748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:15.490744965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1739,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 141 251 251 234 T T F <--- 1 020-039 149 149 149 250 250 250 F F F 2 040-059 148 148 148 246 246 246 T T F <--- 3 060-079 150 150 146 251 251 243 F F F <--- 4 080-099 142 142 142 235 235 235 F F F 5 100-119 150 150 146 252 252 243 F F F <--- 6 120-139 145 145 145 242 242 242 F F F 7 140-159 146 146 146 243 243 243 T T F <--- 8 160-179 148 148 148 244 244 244 F F F 9 180-199 150 150 144 252 252 240 F F F <--- 10 200-219 149 149 149 250 250 250 F F F 11 220-239 149 149 149 250 250 250 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 148 148 148 245 245 245 T T F <--- 14 280-299 148 148 148 249 249 249 F F F 15 300-319 148 148 148 249 249 249 F F F 16 320-339 149 149 149 250 250 250 T T F <--- 17 340-359 150 150 149 252 252 250 F F F <--- 18 360-379 150 150 146 252 252 243 F F F <--- 19 380-399 150 150 148 251 251 246 F F F <--- 20 400-419 150 150 148 252 252 246 F F F <--- 21 420-439 150 150 144 252 252 240 F F F <--- 22 440-459 140 140 140 233 233 233 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 146 252 252 243 F F F <--- 26 520-539 145 145 145 242 242 242 F F F 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 148 148 148 247 247 247 F F F Max gen: 150, Max flush: 252 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 253 253 253 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 151 151 151 253 253 253 F F F 3 060-079 150 150 150 251 251 251 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 150 150 150 252 252 252 F F F 6 120-139 145 145 145 242 242 242 F F F 7 140-159 151 151 151 253 253 253 F F F 8 160-179 148 148 148 244 244 244 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 149 149 149 250 250 250 F F F 11 220-239 149 149 149 250 250 250 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 151 151 151 253 253 253 F F F 14 280-299 148 148 148 249 249 249 F F F 15 300-319 148 148 148 249 249 249 F F F 16 320-339 151 151 151 253 253 253 F F F 17 340-359 150 150 150 252 252 252 F F F 18 360-379 150 150 150 252 252 252 F F F 19 380-399 150 150 150 251 251 251 F F F 20 400-419 150 150 150 252 252 252 F F F 21 420-439 150 150 150 252 252 252 F F F 22 440-459 140 140 140 233 233 233 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 150 252 252 252 F F F 26 520-539 145 145 145 242 242 242 F F F 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 148 148 148 247 247 247 F F F Max gen: 151, Max flush: 253 Begin loop 76 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.754420029Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:19.761569248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.761625506Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.768447024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.768715097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.768750856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.768760236Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.768772075Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Crucible 8a8d51e7-20ac-4a18-b69e-e3ab7752abfa has session id: b62b56f9-2935-4364-b579-226e76259990","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769095316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Upstairs opts: Upstairs UUID: 8a8d51e7-20ac-4a18-b69e-e3ab7752abfa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769127265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Crucible stats registered with UUID: 8a8d51e7-20ac-4a18-b69e-e3ab7752abfa","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769137645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:19.769146655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.76929921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.76931573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.76932577Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769487065Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769504965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"8a8d51e7-20ac-4a18-b69e-e3ab7752abfa active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769532354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769543613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769553583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769563353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769573173Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769583312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769596422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"ds_connection connected from Ok(127.0.0.1:38739)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769632461Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"ds_connection connected from Ok(127.0.0.1:57177)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.76965163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"ds_connection connected from Ok(127.0.0.1:41260)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.769771997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770311142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770330551Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770411389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770652572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770662432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[0]R flush_numbers[0..12]: [253, 250, 253, 251, 235, 252, 242, 253, 244, 252, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.77073251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[0]R generation[0..12]: [151, 149, 151, 150, 142, 150, 145, 151, 148, 150, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.7707462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770757769Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[1]R flush_numbers[0..12]: [253, 250, 253, 251, 235, 252, 242, 253, 244, 252, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770768819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[1]R generation[0..12]: [151, 149, 151, 150, 142, 150, 145, 151, 148, 150, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770779549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770790568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[2]R flush_numbers[0..12]: [253, 250, 253, 251, 235, 252, 242, 253, 244, 252, 250, 250]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770801378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[2]R generation[0..12]: [151, 149, 151, 150, 142, 150, 145, 151, 148, 150, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770811998Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770822888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770832457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770842257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Next flush: 254","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770851567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770860896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"":"downstairs","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770870596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770879416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"0","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770890636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"1","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770900335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"client":"2","":"downstairs","session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770910265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770919425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"8a8d51e7-20ac-4a18-b69e-e3ab7752abfa is now active with session: b62b56f9-2935-4364-b579-226e76259990","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770929365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770938104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"session_id":"b62b56f9-2935-4364-b579-226e76259990"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770947204Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Activated sub_volume 8a8d51e7-20ac-4a18-b69e-e3ab7752abfa","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770955174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.770963354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.771319454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} 01/30 Write block 231 len 4096 data: 10 02/30 Read block 176 len 24576 03/30 Write block 285 len 12288 data: 5 6 6 04/30 Write block 298 len 36864 data: 12 11 14 14 11 9 10 9 14 05/30 Write block 104 len 20480 data: 9 9 10 12 14 06/30 Read block 229 len 32768 07/30 Read block 74 len 12288 08/30 Write block 267 len 40960 data: 13 14 16 18 14 12 12 11 11 9 09/30 Write block 451 len 32768 data: 11 10 12 10 11 14 14 13 10/30 Read block 66 len 40960 11/30 Write block 130 len 32768 data: 6 4 5 6 6 6 7 9 12/30 Read block 269 len 40960 13/30 Write block 208 len 8192 data: 14 15 14/30 Write block 317 len 16384 data: 12 13 13 10 15/30 Write block 420 len 32768 data: 14 13 14 14 13 13 14 12 16/30 Write block 597 len 12288 data: 6 6 4 17/30 Read block 232 len 24576 18/30 Write block 204 len 32768 data: 11 12 9 10 15 16 14 12 19/30 Write block 300 len 24576 data: 15 15 12 10 11 10 20/30 Read block 176 len 36864 21/30 Read block 266 len 24576 22/30 Write block 291 len 20480 data: 5 10 11 8 9 23/30 Read block 266 len 36864 24/30 Write block 530 len 8192 data: 10 12 25/30 Write block 56 len 20480 data: 14 14 12 14 11 26/30 Read block 459 len 8192 27/30 Write block 343 len 20480 data: 10 9 10 10 11 28/30 Read block 562 len 28672 29/30 Read block 148 len 32768 30/30 Read block 492 len 8192 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:19.779669859Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:19.779840174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1747,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 253 253 253 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 151 151 151 253 253 253 F T T <--- 3 060-079 150 150 150 251 251 251 F T T <--- 4 080-099 142 142 142 235 235 235 F F F 5 100-119 150 150 150 252 252 252 F T T <--- 6 120-139 145 145 145 242 242 242 F T T <--- 7 140-159 151 151 151 253 253 253 F F F 8 160-179 148 148 148 244 244 244 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 149 149 149 250 250 250 F T T <--- 11 220-239 149 149 149 250 250 250 F T T <--- 12 240-259 149 149 149 250 250 250 F F F 13 260-279 151 151 151 253 253 253 F T T <--- 14 280-299 148 148 148 249 249 249 F T T <--- 15 300-319 148 148 148 249 249 249 F T T <--- 16 320-339 151 151 151 253 253 253 F T T <--- 17 340-359 150 150 150 252 252 252 F T T <--- 18 360-379 150 150 150 252 252 252 F F F 19 380-399 150 150 150 251 251 251 F F F 20 400-419 150 150 150 252 252 252 F F F 21 420-439 150 150 150 252 252 252 F T T <--- 22 440-459 140 140 140 233 233 233 F T T <--- 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 150 252 252 252 F F F 26 520-539 145 145 145 242 242 242 F T T <--- 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 148 148 148 247 247 247 F T T <--- Max gen: 151, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 253 253 253 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 152 152 152 254 254 254 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 152 152 152 254 254 254 F F F 6 120-139 152 152 152 254 254 254 F F F 7 140-159 151 151 151 253 253 253 F F F 8 160-179 148 148 148 244 244 244 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 152 152 152 254 254 254 F F F 11 220-239 152 152 152 254 254 254 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 152 152 152 254 254 254 F F F 16 320-339 152 152 152 254 254 254 F F F 17 340-359 152 152 152 254 254 254 F F F 18 360-379 150 150 150 252 252 252 F F F 19 380-399 150 150 150 251 251 251 F F F 20 400-419 150 150 150 252 252 252 F F F 21 420-439 152 152 152 254 254 254 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 150 252 252 252 F F F 26 520-539 152 152 152 254 254 254 F F F 27 540-559 148 148 148 245 245 245 F F F 28 560-579 142 142 142 235 235 235 F F F 29 580-599 152 152 152 254 254 254 F F F Max gen: 152, Max flush: 254 Begin loop 77 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.054854829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063170445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063462607Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063491826Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063500416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063514565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Crucible b1131c26-08b3-4aaa-9a0b-c6b66686d43e has session id: 39212b2d-82e1-4f8e-8148-abc31f559a4d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063834426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Upstairs opts: Upstairs UUID: b1131c26-08b3-4aaa-9a0b-c6b66686d43e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063857176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Crucible stats registered with UUID: b1131c26-08b3-4aaa-9a0b-c6b66686d43e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.063867165Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:24.063876335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064021171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064037221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.06404729Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064174207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064191716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"b1131c26-08b3-4aaa-9a0b-c6b66686d43e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064220275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064229795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064239665Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064249695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064259594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064271854Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064281924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45239)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064441439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:61184)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064471228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:57424)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.064482888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065027983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065064292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065076431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065320584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065340764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[0]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 254, 254, 253, 244, 252, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065417772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[0]R generation[0..12]: [151, 149, 152, 152, 142, 152, 152, 151, 148, 150, 152, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065429461Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065445711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[1]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 254, 254, 253, 244, 252, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065456541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[1]R generation[0..12]: [151, 149, 152, 152, 142, 152, 152, 151, 148, 150, 152, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.06546709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.06548169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[2]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 254, 254, 253, 244, 252, 254, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.06549252Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[2]R generation[0..12]: [151, 149, 152, 152, 142, 152, 152, 151, 148, 150, 152, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065503189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065514229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Max found gen is 153","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065523649Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Generation requested: 154 >= found:153","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065533448Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065542848Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065552308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"":"downstairs","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065562088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065570957Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"0","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065580617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"1","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065590397Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"client":"2","":"downstairs","session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065600287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065609496Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"b1131c26-08b3-4aaa-9a0b-c6b66686d43e is now active with session: 39212b2d-82e1-4f8e-8148-abc31f559a4d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065619746Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065630546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065638096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"session_id":"39212b2d-82e1-4f8e-8148-abc31f559a4d"} {"msg":"Activated sub_volume b1131c26-08b3-4aaa-9a0b-c6b66686d43e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065647225Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.065655435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.066011385Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} 01/30 Write block 112 len 8192 data: 14 16 02/30 Write block 593 len 12288 data: 10 10 7 03/30 Write block 128 len 32768 data: 8 6 7 5 6 7 7 7 04/30 Read block 305 len 12288 05/30 Flush 06/30 Write block 210 len 20480 data: 15 13 14 12 10 07/30 Read block 468 len 40960 08/30 Write block 302 len 16384 data: 13 11 12 11 09/30 Read block 233 len 24576 10/30 Read block 24 len 8192 11/30 Write block 580 len 8192 data: 8 11 12/30 Write block 523 len 8192 data: 9 9 13/30 Read block 233 len 8192 14/30 Write block 172 len 20480 data: 11 12 9 11 11 15/30 Write block 582 len 32768 data: 14 14 16 14 14 13 15 11 16/30 Write block 558 len 24576 data: 12 12 14 14 14 13 17/30 Read block 133 len 36864 18/30 Read block 347 len 4096 19/30 Read block 157 len 24576 20/30 Read block 591 len 8192 21/30 Read block 243 len 40960 22/30 Read block 434 len 8192 23/30 Write block 344 len 24576 data: 10 11 11 12 12 10 24/30 Read block 458 len 8192 25/30 Read block 265 len 24576 26/30 Read block 519 len 4096 27/30 Write block 402 len 16384 data: 14 11 12 11 28/30 Write block 333 len 8192 data: 9 10 29/30 Read block 528 len 20480 30/30 Read block 213 len 32768 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Write 3 Done Sent Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) None Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:24.073744598Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:24.073930312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1755,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 253 253 253 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 152 152 152 254 254 254 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 154 152 154 255 254 255 F F F <--- 6 120-139 154 152 154 255 254 255 F F F <--- 7 140-159 151 151 151 253 253 253 F F F 8 160-179 148 148 148 244 244 244 T F T <--- 9 180-199 150 150 150 252 252 252 F F F 10 200-219 152 152 152 254 254 254 T F T <--- 11 220-239 152 152 152 254 254 254 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 152 152 152 254 254 254 T F T <--- 16 320-339 152 152 152 254 254 254 T F T <--- 17 340-359 152 152 152 254 254 254 T F T <--- 18 360-379 150 150 150 252 252 252 F F F 19 380-399 150 150 150 251 251 251 F F F 20 400-419 150 150 150 252 252 252 T F T <--- 21 420-439 152 152 152 254 254 254 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 150 252 252 252 F F F 26 520-539 152 152 152 254 254 254 T F T <--- 27 540-559 148 148 148 245 245 245 T F T <--- 28 560-579 142 142 142 235 235 235 T F T <--- 29 580-599 154 152 154 255 254 255 T F T <--- Max gen: 154, Max flush: 255 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 253 253 253 F F F 1 020-039 149 149 149 250 250 250 F F F 2 040-059 152 152 152 254 254 254 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 154 154 154 255 255 255 F F F 6 120-139 154 154 154 255 255 255 F F F 7 140-159 151 151 151 253 253 253 F F F 8 160-179 155 155 155 256 256 256 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 152 152 152 254 254 254 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 155 155 155 256 256 256 F F F 16 320-339 155 155 155 256 256 256 F F F 17 340-359 155 155 155 256 256 256 F F F 18 360-379 150 150 150 252 252 252 F F F 19 380-399 150 150 150 251 251 251 F F F 20 400-419 155 155 155 256 256 256 F F F 21 420-439 152 152 152 254 254 254 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 146 146 146 243 243 243 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 150 150 150 252 252 252 F F F 26 520-539 155 155 155 256 256 256 F F F 27 540-559 155 155 155 256 256 256 F F F 28 560-579 155 155 155 256 256 256 F F F 29 580-599 155 155 155 256 256 256 F F F Max gen: 155, Max flush: 256 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.334747817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.342892158Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343203139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343229118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343237588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343249378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Crucible c1bd90b0-3a95-4388-b9be-a48c0ec619f5 has session id: ac7b283e-0dc5-4890-8aa1-7166ea7240e4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343614928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Upstairs opts: Upstairs UUID: c1bd90b0-3a95-4388-b9be-a48c0ec619f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343644617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Crucible stats registered with UUID: c1bd90b0-3a95-4388-b9be-a48c0ec619f5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343655586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:28.343667406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343826452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343843211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343853481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343977527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.343993557Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"c1bd90b0-3a95-4388-b9be-a48c0ec619f5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344027266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344036716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344046725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344056645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344066135Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344077894Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344087924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:37449)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344161842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:62415)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344187251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:42844)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344198911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.344923481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.34494662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345037058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345249472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345273351Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[0]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 255, 255, 253, 256, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.34532033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[0]R generation[0..12]: [151, 149, 152, 152, 142, 154, 154, 151, 155, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345332339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345343619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[1]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 255, 255, 253, 256, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345354249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[1]R generation[0..12]: [151, 149, 152, 152, 142, 154, 154, 151, 155, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345364938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345375948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[2]R flush_numbers[0..12]: [253, 250, 254, 254, 235, 255, 255, 253, 256, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345386608Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[2]R generation[0..12]: [151, 149, 152, 152, 142, 154, 154, 151, 155, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345397167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345408347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345421067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345430896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Next flush: 257","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345440236Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345449726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"":"downstairs","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345459476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345468275Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"0","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345478175Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"1","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345487975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"client":"2","":"downstairs","session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345497845Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345506884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"c1bd90b0-3a95-4388-b9be-a48c0ec619f5 is now active with session: ac7b283e-0dc5-4890-8aa1-7166ea7240e4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345516874Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345525764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"session_id":"ac7b283e-0dc5-4890-8aa1-7166ea7240e4"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345534714Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Activated sub_volume c1bd90b0-3a95-4388-b9be-a48c0ec619f5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345542693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345550893Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.345882734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} 01/30 Write block 42 len 20480 data: 22 23 22 21 23 02/30 Write block 589 len 8192 data: 12 10 03/30 Read block 105 len 36864 04/30 Read block 402 len 36864 05/30 Read block 114 len 4096 06/30 Write block 31 len 12288 data: 15 14 13 07/30 Write block 513 len 8192 data: 10 11 08/30 Write block 358 len 40960 data: 12 13 13 13 10 10 11 15 17 16 09/30 Write block 545 len 12288 data: 7 7 10 10/30 Write block 114 len 32768 data: 13 10 13 17 17 20 19 19 11/30 Read block 326 len 4096 12/30 Flush 13/30 Write block 539 len 24576 data: 10 12 11 9 11 9 14/30 Read block 214 len 16384 15/30 Write block 521 len 36864 data: 14 11 10 10 11 10 11 10 9 16/30 Read block 478 len 16384 17/30 Read block 44 len 12288 18/30 Write block 5 len 40960 data: 8 10 13 13 11 10 12 12 12 14 19/30 Read block 312 len 16384 20/30 Read block 124 len 8192 21/30 Write block 472 len 8192 data: 11 12 22/30 Write block 419 len 20480 data: 11 15 14 15 15 23/30 Read block 41 len 20480 24/30 Write block 158 len 20480 data: 6 7 9 11 13 25/30 Flush 26/30 Flush 27/30 Write block 380 len 16384 data: 12 14 13 13 28/30 Read block 119 len 36864 29/30 Read block 200 len 36864 30/30 Write block 155 len 16384 data: 10 10 7 7 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 2 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:28.354630938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:28.354836472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1763,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 151 258 258 253 F F F <--- 1 020-039 156 156 149 257 257 250 F F F <--- 2 040-059 156 156 152 257 257 254 F F T <--- 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 156 156 154 257 257 255 F F F <--- 6 120-139 156 156 154 257 257 255 F F F <--- 7 140-159 156 156 151 258 258 253 T T F <--- 8 160-179 156 156 155 258 258 256 F F F <--- 9 180-199 150 150 150 252 252 252 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 152 152 152 254 254 254 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 155 155 155 256 256 256 F F F 16 320-339 155 155 155 256 256 256 F F F 17 340-359 156 156 155 257 257 256 F F F <--- 18 360-379 156 156 150 257 257 252 F F F <--- 19 380-399 150 150 150 251 251 251 T T F <--- 20 400-419 156 156 155 258 258 256 F F F <--- 21 420-439 156 156 152 258 258 254 F F F <--- 22 440-459 152 152 152 254 254 254 F F F 23 460-479 156 156 146 258 258 243 F F F <--- 24 480-499 148 148 148 249 249 249 F F F 25 500-519 156 156 150 257 257 252 F F F <--- 26 520-539 156 156 155 258 258 256 F F F <--- 27 540-559 156 156 155 258 258 256 F F F <--- 28 560-579 155 155 155 256 256 256 F F F 29 580-599 156 156 155 257 257 256 F F F <--- Max gen: 156, Max flush: 258 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 156 156 156 257 257 257 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 142 142 235 235 235 F F F 5 100-119 156 156 156 257 257 257 F F F 6 120-139 156 156 156 257 257 257 F F F 7 140-159 157 157 157 259 259 259 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 152 152 152 254 254 254 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 155 155 155 256 256 256 F F F 16 320-339 155 155 155 256 256 256 F F F 17 340-359 156 156 156 257 257 257 F F F 18 360-379 156 156 156 257 257 257 F F F 19 380-399 157 157 157 259 259 259 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 156 156 156 258 258 258 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 156 156 156 258 258 258 F F F 24 480-499 148 148 148 249 249 249 F F F 25 500-519 156 156 156 257 257 257 F F F 26 520-539 156 156 156 258 258 258 F F F 27 540-559 156 156 156 258 258 258 F F F 28 560-579 155 155 155 256 256 256 F F F 29 580-599 156 156 156 257 257 257 F F F Max gen: 157, Max flush: 259 Begin loop 79 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.617487425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:32.624662113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.624717732Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.631638217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.63189829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.631923449Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.631931869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.631943618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Crucible 075fcd91-7373-4b8e-8761-cd9ee2b77cf4 has session id: 628d3983-2600-45d5-8bea-cd9745a05bb2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632292849Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Upstairs opts: Upstairs UUID: 075fcd91-7373-4b8e-8761-cd9ee2b77cf4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632321658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Crucible stats registered with UUID: 075fcd91-7373-4b8e-8761-cd9ee2b77cf4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632332778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:32.632342217Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632507263Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632539692Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632550921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632660398Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632670328Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"075fcd91-7373-4b8e-8761-cd9ee2b77cf4 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632686388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632695347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632705067Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632714687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632724277Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632734276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632744096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:52724)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632788145Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:60464)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632813064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:60965)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.632847163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633517054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633530154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633541244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633826666Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633836355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[0]R flush_numbers[0..12]: [258, 257, 257, 254, 235, 257, 257, 259, 258, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633872104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[0]R generation[0..12]: [156, 156, 156, 152, 142, 156, 156, 157, 156, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633883454Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633894514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[1]R flush_numbers[0..12]: [258, 257, 257, 254, 235, 257, 257, 259, 258, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633905313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[1]R generation[0..12]: [156, 156, 156, 152, 142, 156, 156, 157, 156, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633915983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633926923Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[2]R flush_numbers[0..12]: [258, 257, 257, 254, 235, 257, 257, 259, 258, 252, 256, 254]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633937522Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[2]R generation[0..12]: [156, 156, 156, 152, 142, 156, 156, 157, 156, 150, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633948222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633959242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633968872Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633978741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Next flush: 260","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633988081Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.633999621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"":"downstairs","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.63400965Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.63401868Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"0","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.63402834Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"1","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.63403805Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"client":"2","":"downstairs","session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634047949Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634057039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"075fcd91-7373-4b8e-8761-cd9ee2b77cf4 is now active with session: 628d3983-2600-45d5-8bea-cd9745a05bb2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634067009Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634075979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"session_id":"628d3983-2600-45d5-8bea-cd9745a05bb2"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634085098Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Activated sub_volume 075fcd91-7373-4b8e-8761-cd9ee2b77cf4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634093428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634101488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.634434378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} 01/30 Write block 557 len 8192 data: 14 13 02/30 Read block 580 len 12288 03/30 Write block 301 len 8192 data: 16 14 04/30 Write block 149 len 40960 data: 15 15 17 16 15 9 11 11 8 8 05/30 Flush 06/30 Write block 101 len 40960 data: 9 10 7 10 10 11 13 15 14 13 07/30 Write block 322 len 20480 data: 11 14 17 16 20 08/30 Read block 239 len 4096 09/30 Read block 476 len 28672 10/30 Read block 49 len 20480 11/30 Read block 75 len 20480 12/30 Write block 97 len 12288 data: 8 6 7 13/30 Read block 25 len 28672 14/30 Read block 421 len 16384 15/30 Write block 508 len 12288 data: 11 9 8 16/30 Read block 370 len 20480 17/30 Flush 18/30 Write block 416 len 20480 data: 8 11 10 12 16 19/30 Write block 309 len 4096 data: 13 20/30 Write block 555 len 4096 data: 13 21/30 Read block 471 len 4096 22/30 Write block 541 len 4096 data: 12 23/30 Write block 577 len 4096 data: 12 24/30 Read block 386 len 20480 25/30 Read block 461 len 40960 26/30 Write block 476 len 20480 data: 13 9 8 7 7 27/30 Write block 339 len 32768 data: 10 12 12 9 11 11 12 12 28/30 Write block 227 len 40960 data: 10 11 10 10 11 8 10 12 12 10 29/30 Read block 54 len 4096 30/30 Read block 586 len 36864 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 2 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 5 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:32.64219568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:32.642376435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1771,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 156 156 156 257 257 257 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 142 158 158 235 261 261 F F F <--- 5 100-119 156 158 158 257 261 261 F F F <--- 6 120-139 156 156 156 257 257 257 F F F 7 140-159 157 158 158 259 260 260 F F F <--- 8 160-179 156 156 156 258 258 258 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 152 152 152 254 254 254 F T T <--- 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 155 158 158 256 260 260 F T T <--- 16 320-339 155 158 158 256 261 261 F T T <--- 17 340-359 156 156 156 257 257 257 F T T <--- 18 360-379 156 156 156 257 257 257 F F F 19 380-399 157 157 157 259 259 259 F F F 20 400-419 156 156 156 258 258 258 F T T <--- 21 420-439 156 156 156 258 258 258 F T T <--- 22 440-459 152 152 152 254 254 254 F F F 23 460-479 156 156 156 258 258 258 F T T <--- 24 480-499 148 148 148 249 249 249 F T T <--- 25 500-519 156 158 158 257 261 261 F F F <--- 26 520-539 156 156 156 258 258 258 F F F 27 540-559 156 158 158 258 260 260 T T T <--- 28 560-579 155 155 155 256 256 256 F T T <--- 29 580-599 156 156 156 257 257 257 F F F Max gen: 158, Max flush: 261 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 156 156 156 257 257 257 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 158 158 261 261 261 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 156 156 156 257 257 257 F F F 7 140-159 158 158 158 260 260 260 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 150 150 150 252 252 252 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 159 159 159 262 262 262 F F F 12 240-259 149 149 149 250 250 250 F F F 13 260-279 152 152 152 254 254 254 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 159 159 159 262 262 262 F F F 16 320-339 159 159 159 262 262 262 F F F 17 340-359 159 159 159 262 262 262 F F F 18 360-379 156 156 156 257 257 257 F F F 19 380-399 157 157 157 259 259 259 F F F 20 400-419 159 159 159 262 262 262 F F F 21 420-439 159 159 159 262 262 262 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 159 159 159 262 262 262 F F F 24 480-499 159 159 159 262 262 262 F F F 25 500-519 158 158 158 261 261 261 F F F 26 520-539 156 156 156 258 258 258 F F F 27 540-559 159 159 159 262 262 262 F F F 28 560-579 159 159 159 262 262 262 F F F 29 580-599 156 156 156 257 257 257 F F F Max gen: 159, Max flush: 262 Begin loop 80 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.911593033Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.919908259Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920203421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.92023289Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.9202458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920258039Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Crucible 29142872-8140-4447-99c3-386e1c7f16b5 has session id: 9cb890cf-2850-4b0f-ae29-a488314a7e3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920624409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Upstairs opts: Upstairs UUID: 29142872-8140-4447-99c3-386e1c7f16b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920654368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Crucible stats registered with UUID: 29142872-8140-4447-99c3-386e1c7f16b5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920665658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:36.920675058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920804034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920821483Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920831573Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.920985459Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921014688Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"29142872-8140-4447-99c3-386e1c7f16b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921022948Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921031617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921041357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921050947Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921062907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921073086Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921106975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55777)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921128775Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63128)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921140104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64974)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.92128361Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921807766Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921874364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.921886143Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922135176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922166286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[0]R flush_numbers[0..12]: [258, 257, 257, 254, 261, 261, 257, 260, 258, 252, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922214394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[0]R generation[0..12]: [156, 156, 156, 152, 158, 158, 156, 158, 156, 150, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922226334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922237684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[1]R flush_numbers[0..12]: [258, 257, 257, 254, 261, 261, 257, 260, 258, 252, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922251393Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[1]R generation[0..12]: [156, 156, 156, 152, 158, 158, 156, 158, 156, 150, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922262103Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922273073Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[2]R flush_numbers[0..12]: [258, 257, 257, 254, 261, 261, 257, 260, 258, 252, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922283792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[2]R generation[0..12]: [156, 156, 156, 152, 158, 158, 156, 158, 156, 150, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922294382Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922305582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922315061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922324851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Next flush: 263","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922334181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922343621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"":"downstairs","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.92235329Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.92236228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"0","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922372Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"1","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.92238189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"client":"2","":"downstairs","session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922391629Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922402459Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"29142872-8140-4447-99c3-386e1c7f16b5 is now active with session: 9cb890cf-2850-4b0f-ae29-a488314a7e3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922412509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922421358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"session_id":"9cb890cf-2850-4b0f-ae29-a488314a7e3e"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922430628Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Activated sub_volume 29142872-8140-4447-99c3-386e1c7f16b5","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922438748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922446978Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.922792898Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} 01/30 Write block 302 len 24576 data: 15 12 13 12 15 14 02/30 Write block 372 len 24576 data: 13 10 11 13 14 15 03/30 Write block 301 len 28672 data: 17 16 13 14 13 16 15 04/30 Read block 397 len 8192 05/30 Write block 35 len 8192 data: 15 13 06/30 Write block 138 len 24576 data: 9 8 7 10 9 9 07/30 Read block 572 len 4096 08/30 Read block 405 len 20480 09/30 Write block 507 len 8192 data: 9 12 10/30 Write block 403 len 8192 data: 12 13 11/30 Read block 109 len 4096 12/30 Read block 578 len 36864 13/30 Write block 548 len 12288 data: 10 13 16 14/30 Read block 542 len 24576 15/30 Write block 302 len 32768 data: 17 14 15 14 17 16 13 14 16/30 Read block 495 len 20480 17/30 Write block 563 len 12288 data: 14 15 12 18/30 Write block 189 len 40960 data: 14 13 13 12 16 16 15 15 15 16 19/30 Read block 391 len 40960 20/30 Read block 407 len 20480 21/30 Read block 585 len 12288 22/30 Write block 257 len 16384 data: 8 8 10 8 23/30 Read block 541 len 4096 24/30 Write block 249 len 20480 data: 18 21 19 16 15 25/30 Write block 379 len 24576 data: 13 13 15 14 14 14 26/30 Flush 27/30 Write block 485 len 16384 data: 8 7 8 8 28/30 Write block 421 len 36864 data: 15 16 16 14 14 15 13 11 10 29/30 Read block 264 len 16384 30/30 Read block 598 len 4096 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 6 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:36.930720015Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:36.93089771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1779,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 160 160 156 263 263 257 F F F <--- 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 158 158 261 261 261 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 160 160 156 263 263 257 F F F <--- 7 140-159 160 160 158 263 263 260 F F F <--- 8 160-179 156 156 156 258 258 258 F F F 9 180-199 160 160 150 263 263 252 F F F <--- 10 200-219 155 155 155 256 256 256 F F F 11 220-239 159 159 159 262 262 262 F F F 12 240-259 160 160 149 263 263 250 F F F <--- 13 260-279 160 160 152 263 263 254 F F F <--- 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 159 263 263 262 F F T <--- 16 320-339 159 159 159 262 262 262 F F F 17 340-359 159 159 159 262 262 262 F F F 18 360-379 160 160 156 263 263 257 F F F <--- 19 380-399 160 160 157 263 263 259 F F F <--- 20 400-419 160 160 159 263 263 262 F F F <--- 21 420-439 159 159 159 262 262 262 T T F <--- 22 440-459 152 152 152 254 254 254 F F F 23 460-479 159 159 159 262 262 262 F F F 24 480-499 159 159 159 262 262 262 T T F <--- 25 500-519 160 160 158 263 263 261 F F F <--- 26 520-539 156 156 156 258 258 258 F F F 27 540-559 160 160 159 263 263 262 F F F <--- 28 560-579 160 160 159 263 263 262 F F F <--- 29 580-599 156 156 156 257 257 257 F F F Max gen: 160, Max flush: 263 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 160 160 160 263 263 263 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 158 158 261 261 261 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 160 160 160 263 263 263 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 160 160 160 263 263 263 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 159 159 159 262 262 262 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 160 160 160 263 263 263 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F F F 16 320-339 159 159 159 262 262 262 F F F 17 340-359 159 159 159 262 262 262 F F F 18 360-379 160 160 160 263 263 263 F F F 19 380-399 160 160 160 263 263 263 F F F 20 400-419 160 160 160 263 263 263 F F F 21 420-439 161 161 161 264 264 264 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 159 159 159 262 262 262 F F F 24 480-499 161 161 161 264 264 264 F F F 25 500-519 160 160 160 263 263 263 F F F 26 520-539 156 156 156 258 258 258 F F F 27 540-559 160 160 160 263 263 263 F F F 28 560-579 160 160 160 263 263 263 F F F 29 580-599 156 156 156 257 257 257 F F F Max gen: 161, Max flush: 264 Begin loop 81 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.192882602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201238147Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201542238Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201569697Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201578377Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201590097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Crucible b9ec5f3d-1620-44e4-86f3-30a46efd5d89 has session id: 160c8e2e-1ac5-4e87-b980-7098701ec8e0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201918138Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Upstairs opts: Upstairs UUID: b9ec5f3d-1620-44e4-86f3-30a46efd5d89, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201967446Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Crucible stats registered with UUID: b9ec5f3d-1620-44e4-86f3-30a46efd5d89","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.201980006Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:41.201989126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202140821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202157941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202168011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202296927Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202312527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"b9ec5f3d-1620-44e4-86f3-30a46efd5d89 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202335806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202345096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202354875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202364625Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202374305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202384595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202394434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39744)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202511991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:57700)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.20253959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:57938)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.202599548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203313888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203352097Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203363857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203654609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203700788Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[0]R flush_numbers[0..12]: [258, 263, 257, 254, 261, 261, 263, 263, 258, 263, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203737566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[0]R generation[0..12]: [156, 160, 156, 152, 158, 158, 160, 160, 156, 160, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203748856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203759966Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[1]R flush_numbers[0..12]: [258, 263, 257, 254, 261, 261, 263, 263, 258, 263, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203770736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[1]R generation[0..12]: [156, 160, 156, 152, 158, 158, 160, 160, 156, 160, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203781285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203792255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[2]R flush_numbers[0..12]: [258, 263, 257, 254, 261, 261, 263, 263, 258, 263, 256, 262]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203802855Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[2]R generation[0..12]: [156, 160, 156, 152, 158, 158, 160, 160, 156, 160, 155, 159]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203815884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203826784Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203836414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203846113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Next flush: 265","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203855453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203864973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"":"downstairs","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203874853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203883652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"0","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203893432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"1","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203903042Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"client":"2","":"downstairs","session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203912912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203922031Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"b9ec5f3d-1620-44e4-86f3-30a46efd5d89 is now active with session: 160c8e2e-1ac5-4e87-b980-7098701ec8e0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203932141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203941301Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.203948941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"session_id":"160c8e2e-1ac5-4e87-b980-7098701ec8e0"} {"msg":"Activated sub_volume b9ec5f3d-1620-44e4-86f3-30a46efd5d89","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.20395823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.20396687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.204297451Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} 01/30 Write block 522 len 40960 data: 12 11 11 12 11 12 11 10 11 13 02/30 Read block 218 len 40960 03/30 Read block 387 len 24576 04/30 Read block 78 len 8192 05/30 Write block 587 len 16384 data: 14 16 13 11 06/30 Write block 410 len 4096 data: 8 07/30 Read block 506 len 16384 08/30 Write block 578 len 20480 data: 13 11 9 12 15 09/30 Write block 356 len 36864 data: 10 8 13 14 14 14 11 11 12 10/30 Write block 325 len 8192 data: 17 21 11/30 Read block 543 len 12288 12/30 Read block 58 len 32768 13/30 Read block 47 len 28672 14/30 Write block 188 len 32768 data: 14 15 14 14 13 17 17 16 15/30 Write block 173 len 20480 data: 13 10 12 12 12 16/30 Write block 351 len 4096 data: 12 17/30 Write block 234 len 4096 data: 13 18/30 Flush 19/30 Write block 433 len 8192 data: 10 10 20/30 Write block 24 len 4096 data: 14 21/30 Read block 475 len 40960 22/30 Read block 226 len 36864 23/30 Read block 53 len 32768 24/30 Write block 320 len 16384 data: 11 12 12 15 25/30 Write block 484 len 36864 data: 10 9 8 9 9 10 12 12 12 26/30 Read block 230 len 28672 27/30 Read block 587 len 32768 28/30 Read block 56 len 28672 29/30 Read block 518 len 24576 30/30 Write block 188 len 20480 data: 15 16 15 15 14 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:41.212977027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:41.213172281Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1787,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 160 160 160 263 263 263 T F T <--- 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 158 158 261 261 261 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 160 160 160 263 263 263 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 162 156 162 265 258 265 F F F <--- 9 180-199 162 160 162 265 263 265 T F T <--- 10 200-219 155 155 155 256 256 256 F F F 11 220-239 162 159 162 265 262 265 F F F <--- 12 240-259 160 160 160 263 263 263 F F F 13 260-279 160 160 160 263 263 263 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F F F 16 320-339 162 159 162 265 262 265 T F T <--- 17 340-359 162 159 162 265 262 265 F F F <--- 18 360-379 162 160 162 265 263 265 F F F <--- 19 380-399 160 160 160 263 263 263 F F F 20 400-419 162 160 162 265 263 265 F F F <--- 21 420-439 161 161 161 264 264 264 T F T <--- 22 440-459 152 152 152 254 254 254 F F F 23 460-479 159 159 159 262 262 262 F F F 24 480-499 161 161 161 264 264 264 T F T <--- 25 500-519 160 160 160 263 263 263 F F F 26 520-539 162 156 162 265 258 265 F T F <--- 27 540-559 160 160 160 263 263 263 F F F 28 560-579 162 160 162 265 263 265 F F F <--- 29 580-599 162 156 162 265 257 265 F F F <--- Max gen: 162, Max flush: 265 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 163 163 163 266 266 266 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 158 158 261 261 261 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 160 160 160 263 263 263 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 162 162 162 265 265 265 F F F 9 180-199 163 163 163 266 266 266 F F F 10 200-219 155 155 155 256 256 256 F F F 11 220-239 162 162 162 265 265 265 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 160 160 160 263 263 263 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F F F 16 320-339 163 163 163 266 266 266 F F F 17 340-359 162 162 162 265 265 265 F F F 18 360-379 162 162 162 265 265 265 F F F 19 380-399 160 160 160 263 263 263 F F F 20 400-419 162 162 162 265 265 265 F F F 21 420-439 163 163 163 266 266 266 F F F 22 440-459 152 152 152 254 254 254 F F F 23 460-479 159 159 159 262 262 262 F F F 24 480-499 163 163 163 266 266 266 F F F 25 500-519 160 160 160 263 263 263 F F F 26 520-539 162 162 162 265 265 265 F F F 27 540-559 160 160 160 263 263 263 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 162 162 162 265 265 265 F F F Max gen: 163, Max flush: 266 Begin loop 82 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.473724283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:45.480882572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.48094145Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.487831517Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488096239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488131548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488140928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488153017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Crucible 6ffe7c66-ac73-4531-9104-ef3ee6b88519 has session id: 7bc54701-1f7a-42e7-a156-23f0e8aafb2e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488455819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Upstairs opts: Upstairs UUID: 6ffe7c66-ac73-4531-9104-ef3ee6b88519, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488501698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Crucible stats registered with UUID: 6ffe7c66-ac73-4531-9104-ef3ee6b88519","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488512287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:45.488521587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488678473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488694122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488704332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488833398Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488852188Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"6ffe7c66-ac73-4531-9104-ef3ee6b88519 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488879347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488888827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488898777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488908576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488918226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488930886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488940905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55149)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.488989954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44609)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.489015883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:41732)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.489027313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.489747863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.489764332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.489780092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490078843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490088413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[0]R flush_numbers[0..12]: [258, 266, 257, 254, 261, 261, 263, 263, 265, 266, 256, 265]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490124882Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[0]R generation[0..12]: [156, 163, 156, 152, 158, 158, 160, 160, 162, 163, 155, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490136282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490147371Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[1]R flush_numbers[0..12]: [258, 266, 257, 254, 261, 261, 263, 263, 265, 266, 256, 265]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490157961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[1]R generation[0..12]: [156, 163, 156, 152, 158, 158, 160, 160, 162, 163, 155, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490168631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490179541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[2]R flush_numbers[0..12]: [258, 266, 257, 254, 261, 261, 263, 263, 265, 266, 256, 265]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.4901903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[2]R generation[0..12]: [156, 163, 156, 152, 158, 158, 160, 160, 162, 163, 155, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.49020131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.49021268Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490222239Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490233879Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Next flush: 267","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490243439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490252798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"":"downstairs","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490262698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490271448Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"0","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490281018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"1","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490290687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"client":"2","":"downstairs","session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490300567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490309767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"6ffe7c66-ac73-4531-9104-ef3ee6b88519 is now active with session: 7bc54701-1f7a-42e7-a156-23f0e8aafb2e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490319687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490328436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"session_id":"7bc54701-1f7a-42e7-a156-23f0e8aafb2e"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490337566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Activated sub_volume 6ffe7c66-ac73-4531-9104-ef3ee6b88519","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490345576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490353806Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.490689296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} 01/30 Write block 123 len 12288 data: 14 10 12 02/30 Write block 238 len 8192 data: 13 16 03/30 Read block 235 len 36864 04/30 Write block 495 len 8192 data: 12 11 05/30 Write block 101 len 20480 data: 10 11 8 11 11 06/30 Write block 267 len 40960 data: 14 15 17 19 15 13 13 12 12 10 07/30 Write block 467 len 32768 data: 7 8 7 10 10 12 13 12 08/30 Read block 251 len 12288 09/30 Read block 155 len 36864 10/30 Read block 541 len 32768 11/30 Flush 12/30 Write block 544 len 20480 data: 10 8 8 11 11 13/30 Read block 18 len 40960 14/30 Read block 96 len 36864 15/30 Write block 95 len 36864 data: 9 10 9 7 8 8 11 12 9 16/30 Read block 588 len 8192 17/30 Write block 175 len 28672 data: 13 13 13 11 12 11 13 18/30 Write block 181 len 40960 data: 14 13 12 14 13 15 16 16 17 16 19/30 Write block 548 len 40960 data: 12 14 17 16 18 17 13 14 14 15 20/30 Write block 515 len 16384 data: 10 12 13 15 21/30 Flush 22/30 Flush 23/30 Write block 191 len 40960 data: 16 15 18 18 17 16 16 17 17 16 24/30 Read block 164 len 20480 25/30 Write block 508 len 28672 data: 13 10 9 10 11 11 12 26/30 Read block 14 len 4096 27/30 Write block 451 len 36864 data: 12 11 13 11 12 15 15 14 10 28/30 Read block 330 len 40960 29/30 Read block 181 len 12288 30/30 Write block 160 len 24576 data: 10 12 14 14 12 12 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 1 1 25 Done 7 29 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:45.500269507Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:45.500438422Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1795,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 163 163 163 266 266 266 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 158 164 164 261 268 268 F F F <--- 5 100-119 158 164 164 261 268 268 T F F <--- 6 120-139 160 164 164 263 267 267 T F F <--- 7 140-159 160 160 160 263 263 263 F F F 8 160-179 162 164 164 265 268 268 F T T <--- 9 180-199 163 164 164 266 268 268 F T T <--- 10 200-219 155 155 155 256 256 256 F T T <--- 11 220-239 162 164 164 265 267 267 T F F <--- 12 240-259 160 160 160 263 263 263 F F F 13 260-279 160 164 164 263 267 267 T F F <--- 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F F F 16 320-339 163 163 163 266 266 266 F F F 17 340-359 162 162 162 265 265 265 F F F 18 360-379 162 162 162 265 265 265 F F F 19 380-399 160 160 160 263 263 263 F F F 20 400-419 162 162 162 265 265 265 F F F 21 420-439 163 163 163 266 266 266 F F F 22 440-459 152 152 152 254 254 254 F T T <--- 23 460-479 159 164 164 262 267 267 T F F <--- 24 480-499 163 164 164 266 267 267 T F F <--- 25 500-519 160 164 164 263 268 268 F T T <--- 26 520-539 162 162 162 265 265 265 F F F 27 540-559 160 164 164 263 268 268 F F F <--- 28 560-579 162 162 162 265 265 265 F F F 29 580-599 162 162 162 265 265 265 F F F Max gen: 164, Max flush: 268 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 163 163 163 266 266 266 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 164 164 164 268 268 268 F F F 5 100-119 164 164 164 268 268 268 F F F 6 120-139 164 164 164 267 267 267 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 165 165 165 269 269 269 F F F 9 180-199 165 165 165 269 269 269 F F F 10 200-219 165 165 165 269 269 269 F F F 11 220-239 164 164 164 267 267 267 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F F F 16 320-339 163 163 163 266 266 266 F F F 17 340-359 162 162 162 265 265 265 F F F 18 360-379 162 162 162 265 265 265 F F F 19 380-399 160 160 160 263 263 263 F F F 20 400-419 162 162 162 265 265 265 F F F 21 420-439 163 163 163 266 266 266 F F F 22 440-459 165 165 165 269 269 269 F F F 23 460-479 164 164 164 267 267 267 F F F 24 480-499 164 164 164 267 267 267 F F F 25 500-519 165 165 165 269 269 269 F F F 26 520-539 162 162 162 265 265 265 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 162 162 162 265 265 265 F F F Max gen: 165, Max flush: 269 Begin loop 83 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.765648373Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:49.772777283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.772832981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.77961704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.779885073Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.779910332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.779918782Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.779930452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Crucible 3a2cb17d-1bca-4066-98ca-44b714882633 has session id: cbffe180-3a4a-4c72-b44b-e99f7d7a504f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780270982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Upstairs opts: Upstairs UUID: 3a2cb17d-1bca-4066-98ca-44b714882633, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780301911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Crucible stats registered with UUID: 3a2cb17d-1bca-4066-98ca-44b714882633","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780312891Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:49.780322051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780463507Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780497016Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780510575Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780600693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780613752Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"3a2cb17d-1bca-4066-98ca-44b714882633 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780637402Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780646671Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780656721Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780666491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780676031Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.78068609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.78069626Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55694)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780729549Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"ds_connection connected from Ok(127.0.0.1:40009)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780744249Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60388)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.780876755Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781339322Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781359251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781460119Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781647243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781663983Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[0]R flush_numbers[0..12]: [258, 266, 257, 254, 268, 268, 267, 263, 269, 269, 269, 267]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.78175506Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[0]R generation[0..12]: [156, 163, 156, 152, 164, 164, 164, 160, 165, 165, 165, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.78176691Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.78177842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[1]R flush_numbers[0..12]: [258, 266, 257, 254, 268, 268, 267, 263, 269, 269, 269, 267]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781789319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[1]R generation[0..12]: [156, 163, 156, 152, 164, 164, 164, 160, 165, 165, 165, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781800379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781811699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[2]R flush_numbers[0..12]: [258, 266, 257, 254, 268, 268, 267, 263, 269, 269, 269, 267]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781822768Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[2]R generation[0..12]: [156, 163, 156, 152, 164, 164, 164, 160, 165, 165, 165, 164]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781833498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781844648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781854307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781864327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Next flush: 270","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781874027Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781883847Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"":"downstairs","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781895546Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781904566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"0","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781914426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"1","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781924525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"client":"2","":"downstairs","session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781934725Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781944085Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"3a2cb17d-1bca-4066-98ca-44b714882633 is now active with session: cbffe180-3a4a-4c72-b44b-e99f7d7a504f","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781954365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781963504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"session_id":"cbffe180-3a4a-4c72-b44b-e99f7d7a504f"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781973004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Activated sub_volume 3a2cb17d-1bca-4066-98ca-44b714882633","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781981084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.781989334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.782326374Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} 01/30 Write block 599 len 4096 data: 5 02/30 Write block 179 len 12288 data: 13 12 15 03/30 Read block 295 len 32768 04/30 Write block 401 len 20480 data: 12 15 13 14 12 05/30 Read block 481 len 28672 06/30 Read block 110 len 16384 07/30 Write block 530 len 16384 data: 12 14 14 12 08/30 Write block 224 len 28672 data: 11 14 15 11 12 11 11 09/30 Read block 583 len 12288 10/30 Read block 544 len 8192 11/30 Write block 98 len 16384 data: 8 9 9 12 12/30 Write block 91 len 40960 data: 5 6 6 9 10 11 10 9 10 10 13/30 Write block 370 len 4096 data: 14 14/30 Read block 297 len 4096 15/30 Read block 106 len 12288 16/30 Read block 377 len 8192 17/30 Write block 359 len 20480 data: 15 15 15 12 12 18/30 Read block 58 len 12288 19/30 Read block 418 len 24576 20/30 Read block 474 len 24576 21/30 Read block 275 len 12288 22/30 Write block 307 len 4096 data: 17 23/30 Write block 580 len 12288 data: 10 13 16 24/30 Write block 494 len 16384 data: 10 13 12 11 25/30 Read block 529 len 8192 26/30 Write block 451 len 8192 data: 13 12 27/30 Read block 511 len 28672 28/30 Write block 371 len 28672 data: 15 14 11 12 14 15 16 29/30 Read block 283 len 20480 30/30 Read block 298 len 16384 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:49.789735426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:49.789915951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1803,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 163 163 163 266 266 266 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 164 164 164 268 268 268 F T T <--- 5 100-119 164 164 164 268 268 268 F T T <--- 6 120-139 164 164 164 267 267 267 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 165 165 165 269 269 269 F T T <--- 9 180-199 165 165 165 269 269 269 F T T <--- 10 200-219 165 165 165 269 269 269 F F F 11 220-239 164 164 164 267 267 267 F T T <--- 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 160 160 160 263 263 263 F T T <--- 16 320-339 163 163 163 266 266 266 F F F 17 340-359 162 162 162 265 265 265 F T T <--- 18 360-379 162 162 162 265 265 265 F T T <--- 19 380-399 160 160 160 263 263 263 F F F 20 400-419 162 162 162 265 265 265 F T T <--- 21 420-439 163 163 163 266 266 266 F F F 22 440-459 165 165 165 269 269 269 F T T <--- 23 460-479 164 164 164 267 267 267 F F F 24 480-499 164 164 164 267 267 267 F T T <--- 25 500-519 165 165 165 269 269 269 F F F 26 520-539 162 162 162 265 265 265 F T T <--- 27 540-559 164 164 164 268 268 268 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 162 162 162 265 265 265 F T T <--- Max gen: 165, Max flush: 269 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 163 163 163 266 266 266 F F F 2 040-059 156 156 156 257 257 257 F F F 3 060-079 152 152 152 254 254 254 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 166 166 166 270 270 270 F F F 6 120-139 164 164 164 267 267 267 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 F F F 9 180-199 166 166 166 270 270 270 F F F 10 200-219 165 165 165 269 269 269 F F F 11 220-239 166 166 166 270 270 270 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 166 166 166 270 270 270 F F F 16 320-339 163 163 163 266 266 266 F F F 17 340-359 166 166 166 270 270 270 F F F 18 360-379 166 166 166 270 270 270 F F F 19 380-399 160 160 160 263 263 263 F F F 20 400-419 166 166 166 270 270 270 F F F 21 420-439 163 163 163 266 266 266 F F F 22 440-459 166 166 166 270 270 270 F F F 23 460-479 164 164 164 267 267 267 F F F 24 480-499 166 166 166 270 270 270 F F F 25 500-519 165 165 165 269 269 269 F F F 26 520-539 166 166 166 270 270 270 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 166 166 166 270 270 270 F F F Max gen: 166, Max flush: 270 Begin loop 84 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.063210364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.071591618Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.07189652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.071921819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.071932919Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.071944888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Crucible b2e36b55-4dcb-4884-9bad-40f549e5edfe has session id: fcb90fef-64e0-450a-b9ce-3cd5df36d92e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072273889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Upstairs opts: Upstairs UUID: b2e36b55-4dcb-4884-9bad-40f549e5edfe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072318678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Crucible stats registered with UUID: b2e36b55-4dcb-4884-9bad-40f549e5edfe","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072329747Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:54.072338997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072507832Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072524802Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072535092Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072660178Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072672068Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"b2e36b55-4dcb-4884-9bad-40f549e5edfe active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072695817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072705177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072714917Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072724336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072735576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072751626Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072761885Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58521)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072807634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"ds_connection connected from Ok(127.0.0.1:59498)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072858423Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55327)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.072879632Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073494175Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073513854Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073540603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073835525Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073879384Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[0]R flush_numbers[0..12]: [258, 266, 257, 254, 270, 270, 267, 263, 270, 270, 269, 270]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073969771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[0]R generation[0..12]: [156, 163, 156, 152, 166, 166, 164, 160, 166, 166, 165, 166]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073981141Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.073992181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[1]R flush_numbers[0..12]: [258, 266, 257, 254, 270, 270, 267, 263, 270, 270, 269, 270]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.07400306Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[1]R generation[0..12]: [156, 163, 156, 152, 166, 166, 164, 160, 166, 166, 165, 166]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.0740171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.07402873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[2]R flush_numbers[0..12]: [258, 266, 257, 254, 270, 270, 267, 263, 270, 270, 269, 270]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074039619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[2]R generation[0..12]: [156, 163, 156, 152, 166, 166, 164, 160, 166, 166, 165, 166]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074050269Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074061389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Max found gen is 167","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074070928Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Generation requested: 168 >= found:167","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074080648Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Next flush: 271","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074090018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074099488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"":"downstairs","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074109487Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074118197Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"0","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074128137Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"1","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074138117Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"client":"2","":"downstairs","session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074148226Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074157566Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"b2e36b55-4dcb-4884-9bad-40f549e5edfe is now active with session: fcb90fef-64e0-450a-b9ce-3cd5df36d92e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074169346Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074178115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"session_id":"fcb90fef-64e0-450a-b9ce-3cd5df36d92e"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074187445Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Activated sub_volume b2e36b55-4dcb-4884-9bad-40f549e5edfe","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074195645Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074203805Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.074571354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} 01/30 Write block 386 len 28672 data: 15 12 14 15 13 12 16 02/30 Read block 13 len 20480 03/30 Flush 04/30 Write block 31 len 40960 data: 16 15 14 16 16 14 15 18 16 17 05/30 Read block 71 len 36864 06/30 Read block 7 len 32768 07/30 Write block 462 len 24576 data: 10 10 8 9 7 8 08/30 Write block 334 len 8192 data: 11 11 09/30 Read block 165 len 12288 10/30 Read block 102 len 4096 11/30 Read block 68 len 28672 12/30 Flush 13/30 Write block 22 len 32768 data: 14 14 15 15 14 14 13 13 14/30 Read block 314 len 28672 15/30 Write block 229 len 32768 data: 12 12 12 9 11 14 13 11 16/30 Flush 17/30 Write block 513 len 40960 data: 12 13 11 13 14 16 17 17 15 13 18/30 Read block 158 len 20480 19/30 Write block 213 len 20480 data: 13 11 12 14 13 20/30 Read block 363 len 36864 21/30 Read block 32 len 12288 22/30 Flush 23/30 Write block 435 len 24576 data: 12 11 10 9 7 7 24/30 Write block 351 len 4096 data: 13 25/30 Write block 206 len 36864 data: 10 11 16 17 16 14 15 14 12 26/30 Write block 368 len 12288 data: 16 15 15 27/30 Write block 201 len 32768 data: 16 14 14 12 13 11 12 17 28/30 Write block 305 len 40960 data: 15 18 18 14 15 13 14 16 14 13 29/30 Write block 77 len 8192 data: 10 10 30/30 Read block 522 len 24576 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Read 5 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:54.084139905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:54.08433273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1811,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 168 163 168 273 266 273 F F F <--- 2 040-059 168 156 168 272 257 272 F F F <--- 3 060-079 152 152 152 254 254 254 T F T <--- 4 080-099 166 166 166 270 270 270 F F F 5 100-119 166 166 166 270 270 270 F F F 6 120-139 164 164 164 267 267 267 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 F F F 9 180-199 166 166 166 270 270 270 F F F 10 200-219 168 165 168 274 269 274 T F T <--- 11 220-239 168 166 168 273 270 273 F F F <--- 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 166 166 166 270 270 270 T F T <--- 16 320-339 168 163 168 272 266 272 F F F <--- 17 340-359 166 166 166 270 270 270 T F T <--- 18 360-379 166 166 166 270 270 270 T F T <--- 19 380-399 168 160 168 271 263 271 F F F <--- 20 400-419 166 166 166 270 270 270 F F F 21 420-439 163 163 163 266 266 266 T F T <--- 22 440-459 166 166 166 270 270 270 T F T <--- 23 460-479 168 164 168 272 267 272 F F F <--- 24 480-499 166 166 166 270 270 270 F F F 25 500-519 168 165 168 274 269 274 F F F <--- 26 520-539 168 166 168 274 270 274 F F F <--- 27 540-559 164 164 164 268 268 268 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 166 166 166 270 270 270 F F F Max gen: 168, Max flush: 274 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 258 258 258 F F F 1 020-039 168 168 168 273 273 273 F F F 2 040-059 168 168 168 272 272 272 F F F 3 060-079 169 169 169 275 275 275 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 166 166 166 270 270 270 F F F 6 120-139 164 164 164 267 267 267 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 F F F 9 180-199 166 166 166 270 270 270 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 168 168 168 273 273 273 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 F F F 14 280-299 152 152 152 254 254 254 F F F 15 300-319 169 169 169 275 275 275 F F F 16 320-339 168 168 168 272 272 272 F F F 17 340-359 169 169 169 275 275 275 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 168 168 168 271 271 271 F F F 20 400-419 166 166 166 270 270 270 F F F 21 420-439 169 169 169 275 275 275 F F F 22 440-459 169 169 169 275 275 275 F F F 23 460-479 168 168 168 272 272 272 F F F 24 480-499 166 166 166 270 270 270 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 162 162 162 265 265 265 F F F 29 580-599 166 166 166 270 270 270 F F F Max gen: 169, Max flush: 275 Begin loop 85 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.348108761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356300881Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356596353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356622372Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356631002Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356642911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Crucible b4ea4a16-6c66-496a-8e79-dda85a4008c4 has session id: 11334499-fc83-4bcc-8849-2f491df17b66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.356970682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Upstairs opts: Upstairs UUID: b4ea4a16-6c66-496a-8e79-dda85a4008c4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357022791Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Crucible stats registered with UUID: b4ea4a16-6c66-496a-8e79-dda85a4008c4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.35703582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:58.35704516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357197636Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357230455Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357241414Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.35738963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.35740402Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"b4ea4a16-6c66-496a-8e79-dda85a4008c4 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357422539Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357431419Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357441049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357450669Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357460128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357470018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357479808Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"ds_connection connected from Ok(127.0.0.1:33850)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357530296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"ds_connection connected from Ok(127.0.0.1:33359)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357541276Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"ds_connection connected from Ok(127.0.0.1:58648)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.357553716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358280355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358299415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358422821Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358636265Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358648055Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[0]R flush_numbers[0..12]: [258, 273, 272, 275, 270, 270, 267, 263, 270, 270, 275, 273]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358733123Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[0]R generation[0..12]: [156, 168, 168, 169, 166, 166, 164, 160, 166, 166, 169, 168]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358744792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358755912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[1]R flush_numbers[0..12]: [258, 273, 272, 275, 270, 270, 267, 263, 270, 270, 275, 273]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358766712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[1]R generation[0..12]: [156, 168, 168, 169, 166, 166, 164, 160, 166, 166, 169, 168]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358777491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358788521Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[2]R flush_numbers[0..12]: [258, 273, 272, 275, 270, 270, 267, 263, 270, 270, 275, 273]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358799361Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[2]R generation[0..12]: [156, 168, 168, 169, 166, 166, 164, 160, 166, 166, 169, 168]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.3588101Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.35882365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.35883355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358843409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358852809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358862339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"":"downstairs","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358872059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358880668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"0","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358890498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"1","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358900198Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"client":"2","":"downstairs","session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358910078Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358919237Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"b4ea4a16-6c66-496a-8e79-dda85a4008c4 is now active with session: 11334499-fc83-4bcc-8849-2f491df17b66","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358929407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358938347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"session_id":"11334499-fc83-4bcc-8849-2f491df17b66"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358947726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Activated sub_volume b4ea4a16-6c66-496a-8e79-dda85a4008c4","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358955786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.358964026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.359283707Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} 01/30 Write block 118 len 8192 data: 18 21 02/30 Read block 78 len 24576 03/30 Flush 04/30 Write block 579 len 12288 data: 12 11 14 05/30 Read block 340 len 20480 06/30 Write block 420 len 16384 data: 17 16 17 17 07/30 Read block 482 len 36864 08/30 Flush 09/30 Write block 7 len 36864 data: 14 14 12 11 13 13 13 15 14 10/30 Read block 253 len 4096 11/30 Write block 126 len 40960 data: 11 11 9 7 8 6 7 8 8 8 12/30 Read block 58 len 40960 13/30 Write block 57 len 16384 data: 15 13 15 12 14/30 Read block 415 len 12288 15/30 Read block 517 len 4096 16/30 Read block 584 len 16384 17/30 Write block 220 len 8192 data: 13 15 18/30 Read block 55 len 20480 19/30 Write block 487 len 20480 data: 10 10 11 13 13 20/30 Read block 168 len 28672 21/30 Flush 22/30 Write block 425 len 32768 data: 15 16 14 12 11 10 14 9 23/30 Write block 458 len 12288 data: 15 11 9 24/30 Write block 24 len 32768 data: 16 16 15 15 14 14 15 17 25/30 Read block 105 len 36864 26/30 Read block 308 len 16384 27/30 Read block 350 len 4096 28/30 Write block 276 len 32768 data: 11 13 13 12 11 10 10 9 29/30 Read block 71 len 8192 30/30 Write block 483 len 40960 data: 11 11 10 9 11 11 12 14 14 13 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 6 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:14:58.369049723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:14:58.369201908Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1819,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 156 170 278 258 278 F F F <--- 1 020-039 168 168 168 273 273 273 T F T <--- 2 040-059 170 168 170 278 272 278 F F F <--- 3 060-079 170 169 170 278 275 278 F F F <--- 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 166 170 276 270 276 F T F <--- 6 120-139 170 164 170 278 267 278 F F F <--- 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 F F F 9 180-199 166 166 166 270 270 270 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 170 168 170 278 273 278 F F F <--- 12 240-259 160 160 160 263 263 263 F F F 13 260-279 164 164 164 267 267 267 T F T <--- 14 280-299 152 152 152 254 254 254 T F T <--- 15 300-319 169 169 169 275 275 275 F F F 16 320-339 168 168 168 272 272 272 F F F 17 340-359 169 169 169 275 275 275 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 168 168 168 271 271 271 F F F 20 400-419 166 166 166 270 270 270 F F F 21 420-439 170 169 170 277 275 277 T F T <--- 22 440-459 169 169 169 275 275 275 T F T <--- 23 460-479 168 168 168 272 272 272 T F T <--- 24 480-499 170 166 170 278 270 278 T F T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 170 162 170 277 265 277 F F F <--- 29 580-599 170 166 170 277 270 277 F F F <--- Max gen: 170, Max flush: 278 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 278 278 278 F F F 1 020-039 171 171 171 279 279 279 F F F 2 040-059 170 170 170 278 278 278 F F F 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 170 170 170 278 278 278 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 F F F 9 180-199 166 166 166 270 270 270 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 170 170 170 278 278 278 F F F 12 240-259 160 160 160 263 263 263 F F F 13 260-279 171 171 171 279 279 279 F F F 14 280-299 171 171 171 279 279 279 F F F 15 300-319 169 169 169 275 275 275 F F F 16 320-339 168 168 168 272 272 272 F F F 17 340-359 169 169 169 275 275 275 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 168 168 168 271 271 271 F F F 20 400-419 166 166 166 270 270 270 F F F 21 420-439 171 171 171 279 279 279 F F F 22 440-459 171 171 171 279 279 279 F F F 23 460-479 171 171 171 279 279 279 F F F 24 480-499 171 171 171 279 279 279 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 170 170 170 277 277 277 F F F Max gen: 171, Max flush: 279 Begin loop 86 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.631931789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.640097059Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.640392991Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64041795Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64042684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64043869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Crucible 048fc049-17f0-4da3-9663-ad4c1defce3e has session id: dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.640833889Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Upstairs opts: Upstairs UUID: 048fc049-17f0-4da3-9663-ad4c1defce3e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.640863718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Crucible stats registered with UUID: 048fc049-17f0-4da3-9663-ad4c1defce3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.640874587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:02.640883767Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641035313Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641067902Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641078862Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641192868Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641205168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"048fc049-17f0-4da3-9663-ad4c1defce3e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641228637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641238077Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641247857Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641257537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641267206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641277326Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641287206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:46744)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641340884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:35039)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641352034Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:52499)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.641362514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642039425Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642061524Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642072814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642381365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642403234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[0]R flush_numbers[0..12]: [278, 279, 278, 278, 270, 276, 278, 263, 270, 270, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642468963Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[0]R generation[0..12]: [170, 171, 170, 170, 166, 170, 170, 160, 166, 166, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642480492Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642491462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[1]R flush_numbers[0..12]: [278, 279, 278, 278, 270, 276, 278, 263, 270, 270, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642502242Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[1]R generation[0..12]: [170, 171, 170, 170, 166, 170, 170, 160, 166, 166, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642512891Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642523741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[2]R flush_numbers[0..12]: [278, 279, 278, 278, 270, 276, 278, 263, 270, 270, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642534461Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[2]R generation[0..12]: [170, 171, 170, 170, 166, 170, 170, 160, 166, 166, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64254516Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64255644Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64256598Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.64257588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642585389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642594899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"":"downstairs","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642606339Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642615368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"0","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642625378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"1","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642635028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"client":"2","":"downstairs","session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642644878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642654127Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"048fc049-17f0-4da3-9663-ad4c1defce3e is now active with session: dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642664047Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642672907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"session_id":"dcbdbcfd-a0a5-4087-b3ad-3f65918a1d85"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642682017Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Activated sub_volume 048fc049-17f0-4da3-9663-ad4c1defce3e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642690296Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642704036Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.642984548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} 01/30 Write block 397 len 24576 data: 16 15 12 13 13 16 02/30 Write block 9 len 8192 data: 13 12 03/30 Write block 120 len 4096 data: 20 04/30 Read block 184 len 24576 05/30 Write block 249 len 40960 data: 19 22 20 17 16 14 11 11 9 9 06/30 Read block 424 len 28672 07/30 Read block 18 len 4096 08/30 Write block 181 len 4096 data: 16 09/30 Write block 289 len 8192 data: 4 4 10/30 Read block 332 len 36864 11/30 Read block 201 len 12288 12/30 Write block 261 len 40960 data: 8 9 9 12 14 12 15 16 18 20 13/30 Write block 41 len 12288 data: 17 23 24 14/30 Read block 586 len 28672 15/30 Read block 181 len 36864 16/30 Write block 353 len 20480 data: 15 11 10 11 9 17/30 Write block 338 len 28672 data: 8 11 13 13 10 12 12 18/30 Flush 19/30 Write block 34 len 12288 data: 17 17 15 20/30 Write block 287 len 28672 data: 7 6 5 5 6 11 12 21/30 Flush 22/30 Write block 32 len 16384 data: 16 15 18 18 23/30 Read block 302 len 36864 24/30 Read block 178 len 24576 25/30 Write block 320 len 24576 data: 12 13 13 16 18 18 26/30 Read block 151 len 20480 27/30 Write block 273 len 28672 data: 14 13 13 12 14 14 13 28/30 Write block 167 len 8192 data: 11 13 29/30 Write block 313 len 4096 data: 15 30/30 Write block 0 len 32768 data: 4 4 6 6 10 9 11 15 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Read 1 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Write 2 Done Done Done false 1009 Acked Read 9 Done Done Done false 1010 Acked Read 3 Done Done Done false 1011 Acked Write 10 Done Done Done false 1012 Acked Write 3 Done Done Done false 1013 Acked Read 7 Done Done Done false 1014 Acked Read 9 Done Done Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 7 Sent Sent Sent false 1027 Acked Write 2 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 4 15 4 23 Done 26 15 26 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:02.65108779Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:02.651287905Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1827,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 170 172 280 278 280 T T T <--- 1 020-039 172 171 172 281 279 281 T F T <--- 2 040-059 172 170 172 280 278 280 F T F <--- 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 170 172 280 278 280 F T F <--- 7 140-159 160 160 160 263 263 263 F F F 8 160-179 166 166 166 270 270 270 T F T <--- 9 180-199 172 166 172 280 270 280 F T F <--- 10 200-219 169 169 169 275 275 275 F F F 11 220-239 170 170 170 278 278 278 F F F 12 240-259 172 160 172 280 263 280 F T F <--- 13 260-279 172 171 172 280 279 280 T T T <--- 14 280-299 172 171 172 281 279 281 F T F <--- 15 300-319 169 169 169 275 275 275 T F T <--- 16 320-339 172 168 172 280 272 280 T F T <--- 17 340-359 172 169 172 280 275 280 F F F <--- 18 360-379 169 169 169 275 275 275 F F F 19 380-399 172 168 172 280 271 280 F T F <--- 20 400-419 172 166 172 280 270 280 F T F <--- 21 420-439 171 171 171 279 279 279 F F F 22 440-459 171 171 171 279 279 279 F F F 23 460-479 171 171 171 279 279 279 F F F 24 480-499 171 171 171 279 279 279 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 170 170 170 277 277 277 F F F Max gen: 172, Max flush: 281 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 282 282 282 F F F 1 020-039 173 173 173 282 282 282 F F F 2 040-059 172 172 172 280 280 280 F F F 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 170 170 170 278 278 278 F F F 12 240-259 172 172 172 280 280 280 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 173 173 173 282 282 282 F F F 16 320-339 173 173 173 282 282 282 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 172 172 172 280 280 280 F F F 20 400-419 172 172 172 280 280 280 F F F 21 420-439 171 171 171 279 279 279 F F F 22 440-459 171 171 171 279 279 279 F F F 23 460-479 171 171 171 279 279 279 F F F 24 480-499 171 171 171 279 279 279 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 F F F 27 540-559 164 164 164 268 268 268 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 170 170 170 277 277 277 F F F Max gen: 173, Max flush: 282 Begin loop 87 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.918141739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.926369757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.926661769Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.926687358Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.926698508Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.926710318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Crucible 659bc9ff-9199-4c88-8063-4a7433625a90 has session id: ec85f000-9668-4adf-af42-bc0f67f44bf8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927063878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Upstairs opts: Upstairs UUID: 659bc9ff-9199-4c88-8063-4a7433625a90, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927095147Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Crucible stats registered with UUID: 659bc9ff-9199-4c88-8063-4a7433625a90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927107757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:06.927116926Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927274282Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927291941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927301941Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927420838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927434457Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"659bc9ff-9199-4c88-8063-4a7433625a90 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927474196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927483726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927493266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927502856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927514615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927524685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"ds_connection connected from Ok(127.0.0.1:47569)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927561224Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927571744Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"ds_connection connected from Ok(127.0.0.1:52933)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927583303Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"ds_connection connected from Ok(127.0.0.1:34022)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.927627802Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928252704Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928273254Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928353102Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928573485Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928615904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[0]R flush_numbers[0..12]: [282, 282, 280, 278, 270, 276, 280, 263, 282, 280, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928668873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[0]R generation[0..12]: [173, 173, 172, 170, 166, 170, 172, 160, 173, 172, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928679992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928691122Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[1]R flush_numbers[0..12]: [282, 282, 280, 278, 270, 276, 280, 263, 282, 280, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928704432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[1]R generation[0..12]: [173, 173, 172, 170, 166, 170, 172, 160, 173, 172, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928715031Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928726041Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[2]R flush_numbers[0..12]: [282, 282, 280, 278, 270, 276, 280, 263, 282, 280, 275, 278]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928736631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[2]R generation[0..12]: [173, 173, 172, 170, 166, 170, 172, 160, 173, 172, 169, 170]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928747231Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.92875823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.92876794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.92877772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Next flush: 283","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928786959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928796449Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"":"downstairs","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928806049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928814739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"0","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928824458Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"1","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928834168Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"client":"2","":"downstairs","session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928843968Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928854778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"659bc9ff-9199-4c88-8063-4a7433625a90 is now active with session: ec85f000-9668-4adf-af42-bc0f67f44bf8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928864647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928873407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"session_id":"ec85f000-9668-4adf-af42-bc0f67f44bf8"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928882447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Activated sub_volume 659bc9ff-9199-4c88-8063-4a7433625a90","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928890497Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.928898596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.929274406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} 01/30 Write block 228 len 8192 data: 13 13 02/30 Read block 139 len 40960 03/30 Read block 535 len 16384 04/30 Write block 230 len 8192 data: 13 13 05/30 Read block 4 len 36864 06/30 Write block 17 len 36864 data: 9 11 13 14 12 15 15 17 17 07/30 Read block 255 len 4096 08/30 Write block 536 len 16384 data: 12 14 11 11 09/30 Write block 595 len 4096 data: 8 10/30 Read block 313 len 36864 11/30 Write block 404 len 32768 data: 15 13 10 11 11 9 9 9 12/30 Read block 581 len 4096 13/30 Read block 107 len 16384 14/30 Write block 316 len 32768 data: 9 13 14 14 13 14 14 17 15/30 Write block 227 len 28672 data: 12 14 14 14 14 10 12 16/30 Read block 327 len 36864 17/30 Write block 539 len 28672 data: 12 13 13 10 12 11 9 18/30 Read block 512 len 24576 19/30 Read block 190 len 36864 20/30 Write block 395 len 36864 data: 17 18 17 16 13 14 14 17 14 21/30 Write block 48 len 36864 data: 20 18 17 14 13 17 16 12 15 22/30 Read block 522 len 4096 23/30 Read block 267 len 24576 24/30 Read block 296 len 32768 25/30 Read block 103 len 28672 26/30 Write block 427 len 4096 data: 15 27/30 Read block 147 len 4096 28/30 Read block 559 len 8192 29/30 Write block 244 len 16384 data: 10 11 15 15 30/30 Read block 201 len 20480 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Write 2 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 2 27 29 Done 30 28 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:06.938182695Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:06.93835772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1835,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 282 282 282 T T F <--- 1 020-039 173 173 173 282 282 282 T T F <--- 2 040-059 172 172 172 280 280 280 T T F <--- 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 170 170 170 278 278 278 T T T 12 240-259 172 172 172 280 280 280 T T F <--- 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 173 173 173 282 282 282 T T F <--- 16 320-339 173 173 173 282 282 282 T T F <--- 17 340-359 172 172 172 280 280 280 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 172 172 172 280 280 280 T T F <--- 20 400-419 172 172 172 280 280 280 T T F <--- 21 420-439 171 171 171 279 279 279 T T F <--- 22 440-459 171 171 171 279 279 279 F F F 23 460-479 171 171 171 279 279 279 F F F 24 480-499 171 171 171 279 279 279 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 168 168 168 274 274 274 T T F <--- 27 540-559 164 164 164 268 268 268 T T F <--- 28 560-579 170 170 170 277 277 277 F F F 29 580-599 170 170 170 277 277 277 T T F <--- Max gen: 173, Max flush: 282 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 174 174 174 283 283 283 F F F 1 020-039 174 174 174 283 283 283 F F F 2 040-059 174 174 174 283 283 283 F F F 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 169 169 169 275 275 275 F F F 11 220-239 174 174 174 283 283 283 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 174 174 174 283 283 283 F F F 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 169 169 169 275 275 275 F F F 19 380-399 174 174 174 283 283 283 F F F 20 400-419 174 174 174 283 283 283 F F F 21 420-439 174 174 174 283 283 283 F F F 22 440-459 171 171 171 279 279 279 F F F 23 460-479 171 171 171 279 279 279 F F F 24 480-499 171 171 171 279 279 279 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 174 174 174 283 283 283 F F F 27 540-559 174 174 174 283 283 283 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 174 174 174 283 283 283 F F F Max gen: 174, Max flush: 283 Begin loop 88 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.200083569Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.208289349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.20860544Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.208640369Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.208650018Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.208661778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Crucible ed078b82-1eb4-4ee0-9df0-3cdca097ab7c has session id: 6ae6f454-1cc8-4d6a-83e9-e8a6047af518","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209055727Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Upstairs opts: Upstairs UUID: ed078b82-1eb4-4ee0-9df0-3cdca097ab7c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209084066Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Crucible stats registered with UUID: ed078b82-1eb4-4ee0-9df0-3cdca097ab7c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209097386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:11.209106696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209257021Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209273611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209283621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209405107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209420687Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"ed078b82-1eb4-4ee0-9df0-3cdca097ab7c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209454866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209464266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209474135Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209483765Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209493565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209503674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209513734Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"ds_connection connected from Ok(127.0.0.1:45995)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209605962Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"ds_connection connected from Ok(127.0.0.1:45464)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209631441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"ds_connection connected from Ok(127.0.0.1:59000)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.209706089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210248803Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210349501Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.21038565Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210561835Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210668432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[0]R flush_numbers[0..12]: [283, 283, 283, 278, 270, 276, 280, 263, 282, 280, 275, 283]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.21072776Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[0]R generation[0..12]: [174, 174, 174, 170, 166, 170, 172, 160, 173, 172, 169, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.21073995Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210751219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[1]R flush_numbers[0..12]: [283, 283, 283, 278, 270, 276, 280, 263, 282, 280, 275, 283]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210761969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[1]R generation[0..12]: [174, 174, 174, 170, 166, 170, 172, 160, 173, 172, 169, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210772509Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210783478Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[2]R flush_numbers[0..12]: [283, 283, 283, 278, 270, 276, 280, 263, 282, 280, 275, 283]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210794258Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[2]R generation[0..12]: [174, 174, 174, 170, 166, 170, 172, 160, 173, 172, 169, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210804878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210818647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Max found gen is 175","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210828177Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Generation requested: 176 >= found:175","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210838197Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Next flush: 284","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210847597Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210857036Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"":"downstairs","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210866916Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210875786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"0","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210885596Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"1","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210895255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"client":"2","":"downstairs","session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210905075Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210914255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"ed078b82-1eb4-4ee0-9df0-3cdca097ab7c is now active with session: 6ae6f454-1cc8-4d6a-83e9-e8a6047af518","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210924134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210932994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"session_id":"6ae6f454-1cc8-4d6a-83e9-e8a6047af518"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210942234Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Activated sub_volume ed078b82-1eb4-4ee0-9df0-3cdca097ab7c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210950494Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.210958614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.211307554Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} 01/30 Flush 02/30 Write block 29 len 32768 data: 15 16 18 17 16 19 19 16 03/30 Read block 533 len 32768 04/30 Read block 7 len 8192 05/30 Read block 166 len 36864 06/30 Read block 118 len 8192 07/30 Read block 181 len 40960 08/30 Write block 81 len 8192 data: 13 13 09/30 Read block 260 len 8192 10/30 Write block 490 len 16384 data: 15 15 14 10 11/30 Read block 243 len 32768 12/30 Write block 215 len 24576 data: 13 15 14 13 12 14 13/30 Write block 6 len 20480 data: 12 16 15 14 13 14/30 Write block 90 len 32768 data: 8 6 7 7 10 11 12 11 15/30 Write block 12 len 24576 data: 14 14 16 15 12 10 16/30 Read block 356 len 36864 17/30 Read block 115 len 12288 18/30 Read block 15 len 8192 19/30 Write block 36 len 20480 data: 17 16 19 17 18 20/30 Write block 464 len 36864 data: 9 10 8 9 9 8 11 11 13 21/30 Read block 118 len 36864 22/30 Write block 442 len 8192 data: 10 10 23/30 Read block 502 len 32768 24/30 Read block 445 len 8192 25/30 Write block 374 len 36864 data: 13 15 16 17 13 14 14 16 15 26/30 Read block 349 len 40960 27/30 Write block 440 len 40960 data: 8 9 11 11 12 17 15 14 13 11 28/30 Write block 364 len 36864 data: 13 16 18 17 17 16 16 16 15 29/30 Write block 89 len 24576 data: 11 9 7 8 8 11 30/30 Read block 331 len 20480 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Sent Done false 1002 Acked Read 8 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 9 Done Sent Done false 1028 Acked Write 6 Done Sent Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 2 31 Done 29 0 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:11.220023089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:11.220192514Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1843,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 174 174 174 283 283 283 T F T <--- 1 020-039 174 174 174 283 283 283 T F T <--- 2 040-059 174 174 174 283 283 283 T F T <--- 3 060-079 170 170 170 278 278 278 F F F 4 080-099 166 166 166 270 270 270 T F T <--- 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 169 169 169 275 275 275 T F T <--- 11 220-239 174 174 174 283 283 283 T F T <--- 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 174 174 174 283 283 283 F F F 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 169 169 169 275 275 275 T F T <--- 19 380-399 174 174 174 283 283 283 T F T <--- 20 400-419 174 174 174 283 283 283 F F F 21 420-439 174 174 174 283 283 283 F F F 22 440-459 171 171 171 279 279 279 T F T <--- 23 460-479 171 171 171 279 279 279 T F T <--- 24 480-499 171 171 171 279 279 279 T F T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 174 174 174 283 283 283 F F F 27 540-559 174 174 174 283 283 283 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 174 174 174 283 283 283 F F F Max gen: 174, Max flush: 283 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 284 284 284 F F F 1 020-039 175 175 175 284 284 284 F F F 2 040-059 175 175 175 284 284 284 F F F 3 060-079 170 170 170 278 278 278 F F F 4 080-099 175 175 175 284 284 284 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 175 175 175 284 284 284 F F F 11 220-239 175 175 175 284 284 284 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 174 174 174 283 283 283 F F F 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 175 175 175 284 284 284 F F F 19 380-399 175 175 175 284 284 284 F F F 20 400-419 174 174 174 283 283 283 F F F 21 420-439 174 174 174 283 283 283 F F F 22 440-459 175 175 175 284 284 284 F F F 23 460-479 175 175 175 284 284 284 F F F 24 480-499 175 175 175 284 284 284 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 174 174 174 283 283 283 F F F 27 540-559 174 174 174 283 283 283 F F F 28 560-579 170 170 170 277 277 277 F F F 29 580-599 174 174 174 283 283 283 F F F Max gen: 175, Max flush: 284 Begin loop 89 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.478073761Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486183193Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486470675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486494854Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486503394Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486515174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Crucible 4d2809a5-088f-4e44-9e2b-845d59dc2cd2 has session id: e3bd54e7-c5b4-4174-8ba2-0a8eacafae30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486809895Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Upstairs opts: Upstairs UUID: 4d2809a5-088f-4e44-9e2b-845d59dc2cd2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486841924Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Crucible stats registered with UUID: 4d2809a5-088f-4e44-9e2b-845d59dc2cd2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.486852834Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:15.486862064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487012Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487028719Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487039049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487187115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487200824Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"4d2809a5-088f-4e44-9e2b-845d59dc2cd2 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487238763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487247943Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487257623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487267272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487276982Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487286942Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487301181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"ds_connection connected from Ok(127.0.0.1:33971)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.48736025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"ds_connection connected from Ok(127.0.0.1:42305)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487371519Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"ds_connection connected from Ok(127.0.0.1:47375)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.487442537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488016661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488045491Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488164757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488329823Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488345592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[0]R flush_numbers[0..12]: [284, 284, 284, 278, 284, 276, 280, 263, 282, 280, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.48842415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[0]R generation[0..12]: [175, 175, 175, 170, 175, 170, 172, 160, 173, 172, 175, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.48843579Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488446989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[1]R flush_numbers[0..12]: [284, 284, 284, 278, 284, 276, 280, 263, 282, 280, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488457699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[1]R generation[0..12]: [175, 175, 175, 170, 175, 170, 172, 160, 173, 172, 175, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488468389Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488479418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[2]R flush_numbers[0..12]: [284, 284, 284, 278, 284, 276, 280, 263, 282, 280, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488489918Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[2]R generation[0..12]: [175, 175, 175, 170, 175, 170, 172, 160, 173, 172, 175, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488500538Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488511567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488520977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Generation requested: 178 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488530877Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Next flush: 285","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488540147Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488549426Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"":"downstairs","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488560906Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488569606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"0","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488579475Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"1","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488589335Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"client":"2","":"downstairs","session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488599505Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488608675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"4d2809a5-088f-4e44-9e2b-845d59dc2cd2 is now active with session: e3bd54e7-c5b4-4174-8ba2-0a8eacafae30","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488618674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488627404Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"session_id":"e3bd54e7-c5b4-4174-8ba2-0a8eacafae30"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488636634Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Activated sub_volume 4d2809a5-088f-4e44-9e2b-845d59dc2cd2","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488644594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-09-24T22:15:15.488652683Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.488935975Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} 01/30 Write block 366 len 12288 data: 19 18 18 02/30 Write block 307 len 16384 data: 19 15 16 14 03/30 Write block 574 len 24576 data: 8 10 11 13 14 13 04/30 Write block 590 len 8192 data: 12 11 05/30 Read block 430 len 28672 06/30 Read block 20 len 16384 07/30 Read block 340 len 32768 08/30 Read block 260 len 32768 09/30 Write block 472 len 36864 data: 14 14 13 12 14 10 9 8 8 10/30 Read block 230 len 20480 11/30 Flush 12/30 Write block 208 len 28672 data: 18 18 17 15 16 15 13 13/30 Write block 430 len 32768 data: 11 15 10 11 11 13 12 11 14/30 Read block 376 len 36864 15/30 Flush 16/30 Write block 460 len 4096 data: 10 17/30 Read block 274 len 40960 18/30 Read block 496 len 40960 19/30 Write block 470 len 32768 data: 12 12 15 15 14 13 15 11 20/30 Write block 130 len 32768 data: 9 7 8 9 9 9 8 10 21/30 Read block 492 len 20480 22/30 Read block 369 len 12288 23/30 Read block 541 len 36864 24/30 Write block 534 len 20480 data: 12 10 13 15 12 25/30 Read block 189 len 24576 26/30 Write block 64 len 32768 data: 8 12 13 13 14 12 13 10 27/30 Read block 289 len 28672 28/30 Read block 251 len 8192 29/30 Read block 539 len 12288 30/30 Read block 214 len 40960 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 2 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 27 28 Done 29 30 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:15.498281753Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:15.498471657Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1851,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 284 284 284 F F F 1 020-039 175 175 175 284 284 284 F F F 2 040-059 175 175 175 284 284 284 F F F 3 060-079 170 170 170 278 278 278 T T F <--- 4 080-099 175 175 175 284 284 284 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 172 172 172 280 280 280 T T F <--- 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 178 178 175 286 286 284 F F F <--- 11 220-239 175 175 175 284 284 284 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 178 178 174 285 285 283 F F T <--- 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 178 178 175 285 285 284 F F T <--- 19 380-399 175 175 175 284 284 284 F F F 20 400-419 174 174 174 283 283 283 F F F 21 420-439 178 178 174 286 286 283 F F F <--- 22 440-459 175 175 175 284 284 284 F F F 23 460-479 178 178 175 285 285 284 T T F <--- 24 480-499 178 178 175 285 285 284 F F F <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 174 174 174 283 283 283 T T F <--- 27 540-559 174 174 174 283 283 283 F F F 28 560-579 178 178 170 285 285 277 F F T <--- 29 580-599 178 178 174 285 285 283 F F F <--- Max gen: 178, Max flush: 286 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 284 284 284 F F F 1 020-039 175 175 175 284 284 284 F F F 2 040-059 175 175 175 284 284 284 F F F 3 060-079 179 179 179 287 287 287 F F F 4 080-099 175 175 175 284 284 284 F F F 5 100-119 170 170 170 276 276 276 F F F 6 120-139 179 179 179 287 287 287 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 178 178 178 286 286 286 F F F 11 220-239 175 175 175 284 284 284 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 172 172 172 281 281 281 F F F 15 300-319 178 178 178 285 285 285 F F F 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 178 178 178 285 285 285 F F F 19 380-399 175 175 175 284 284 284 F F F 20 400-419 174 174 174 283 283 283 F F F 21 420-439 178 178 178 286 286 286 F F F 22 440-459 175 175 175 284 284 284 F F F 23 460-479 179 179 179 287 287 287 F F F 24 480-499 178 178 178 285 285 285 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 179 179 179 287 287 287 F F F 27 540-559 174 174 174 283 283 283 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 179, Max flush: 287 Begin loop 90 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.75118145Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759300211Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759597693Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759621662Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759633082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759644892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Crucible 182e5a0b-db38-4201-9580-20dd5d71fdeb has session id: d6dbdcbe-569d-4a1e-ae79-8a484a0454f7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.759986552Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Upstairs opts: Upstairs UUID: 182e5a0b-db38-4201-9580-20dd5d71fdeb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760025441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Crucible stats registered with UUID: 182e5a0b-db38-4201-9580-20dd5d71fdeb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760036481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:19.76004586Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760193896Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760223815Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760234765Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760340432Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760350332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"182e5a0b-db38-4201-9580-20dd5d71fdeb active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760383631Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760393171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.76040304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.76041283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.76042408Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.76043416Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"ds_connection connected from Ok(127.0.0.1:54223)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760480088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760490698Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"ds_connection connected from Ok(127.0.0.1:60933)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760501088Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"ds_connection connected from Ok(127.0.0.1:46490)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.760557316Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761161959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761179589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761232537Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.7614764Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.76148615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[0]R flush_numbers[0..12]: [284, 284, 284, 287, 284, 276, 287, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761539888Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[0]R generation[0..12]: [175, 175, 175, 179, 175, 170, 179, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761551288Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761562388Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[1]R flush_numbers[0..12]: [284, 284, 284, 287, 284, 276, 287, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761575387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[1]R generation[0..12]: [175, 175, 175, 179, 175, 170, 179, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761586087Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761597107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[2]R flush_numbers[0..12]: [284, 284, 284, 287, 284, 276, 287, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761607617Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[2]R generation[0..12]: [175, 175, 175, 179, 175, 170, 179, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761618176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761629096Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761638436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761648105Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Next flush: 288","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761657405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761666785Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"":"downstairs","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761676375Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761685044Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"0","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761694754Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"1","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761704354Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"client":"2","":"downstairs","session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761714364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761725633Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"182e5a0b-db38-4201-9580-20dd5d71fdeb is now active with session: d6dbdcbe-569d-4a1e-ae79-8a484a0454f7","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761735523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761744233Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"session_id":"d6dbdcbe-569d-4a1e-ae79-8a484a0454f7"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761753272Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Activated sub_volume 182e5a0b-db38-4201-9580-20dd5d71fdeb","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761761172Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.761769222Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.762135992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} 01/30 Write block 131 len 8192 data: 8 9 02/30 Write block 293 len 28672 data: 13 9 10 9 10 13 12 03/30 Read block 17 len 12288 04/30 Read block 105 len 40960 05/30 Write block 406 len 28672 data: 11 12 12 10 10 10 9 06/30 Read block 38 len 20480 07/30 Read block 76 len 20480 08/30 Write block 109 len 36864 data: 15 14 15 15 17 14 11 14 18 09/30 Flush 10/30 Write block 311 len 8192 data: 15 17 11/30 Read block 96 len 24576 12/30 Write block 95 len 36864 data: 12 13 12 10 11 11 13 13 10 13/30 Read block 200 len 12288 14/30 Write block 550 len 40960 data: 18 17 19 18 14 15 15 16 14 13 15/30 Read block 346 len 24576 16/30 Read block 145 len 12288 17/30 Write block 86 len 40960 data: 12 14 13 12 10 8 9 9 12 13 18/30 Flush 19/30 Flush 20/30 Write block 57 len 40960 data: 16 14 16 13 9 7 8 9 13 14 21/30 Read block 218 len 20480 22/30 Read block 209 len 24576 23/30 Write block 1 len 16384 data: 5 7 7 11 24/30 Write block 365 len 20480 data: 17 20 19 19 17 25/30 Read block 414 len 40960 26/30 Read block 516 len 32768 27/30 Read block 144 len 12288 28/30 Read block 232 len 40960 29/30 Read block 312 len 28672 30/30 Read block 452 len 36864 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 21 22 Done 20 21 0 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) Some(JobId(1008)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:19.772284386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:19.772468131Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1859,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 284 284 284 T T F <--- 1 020-039 175 175 175 284 284 284 F F F 2 040-059 175 175 175 284 284 284 T T F <--- 3 060-079 179 179 179 287 287 287 T T F <--- 4 080-099 180 180 175 289 289 284 F F F <--- 5 100-119 180 180 180 289 289 288 F F F <--- 6 120-139 180 180 180 288 288 288 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 178 178 178 286 286 286 F F F 11 220-239 175 175 175 284 284 284 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 180 180 180 288 288 288 F F F 15 300-319 180 180 178 289 289 285 F F F <--- 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 178 178 178 285 285 285 T T F <--- 19 380-399 175 175 175 284 284 284 F F F 20 400-419 180 180 180 288 288 288 F F F 21 420-439 178 178 178 286 286 286 F F F 22 440-459 175 175 175 284 284 284 F F F 23 460-479 179 179 179 287 287 287 F F F 24 480-499 178 178 178 285 285 285 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 179 179 179 287 287 287 F F F 27 540-559 180 180 174 289 289 283 F F F <--- 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 180, Max flush: 289 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 F F F 1 020-039 175 175 175 284 284 284 F F F 2 040-059 181 181 181 290 290 290 F F F 3 060-079 181 181 181 290 290 290 F F F 4 080-099 180 180 180 289 289 289 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 180 180 180 288 288 288 F F F 7 140-159 160 160 160 263 263 263 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 172 172 172 280 280 280 F F F 10 200-219 178 178 178 286 286 286 F F F 11 220-239 175 175 175 284 284 284 F F F 12 240-259 174 174 174 283 283 283 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 180 180 180 288 288 288 F F F 15 300-319 180 180 180 289 289 289 F F F 16 320-339 174 174 174 283 283 283 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 181 181 181 290 290 290 F F F 19 380-399 175 175 175 284 284 284 F F F 20 400-419 180 180 180 288 288 288 F F F 21 420-439 178 178 178 286 286 286 F F F 22 440-459 175 175 175 284 284 284 F F F 23 460-479 179 179 179 287 287 287 F F F 24 480-499 178 178 178 285 285 285 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 179 179 179 287 287 287 F F F 27 540-559 180 180 180 289 289 289 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 181, Max flush: 290 Begin loop 91 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.025970381Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03418869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.034491461Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.0345164Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03452551Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03453726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Crucible b691267b-003e-445d-a2f0-d259556aa502 has session id: 785ca8fe-9dd5-4809-b3ed-8c6b939cf45e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.034863751Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Upstairs opts: Upstairs UUID: b691267b-003e-445d-a2f0-d259556aa502, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03489312Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Crucible stats registered with UUID: b691267b-003e-445d-a2f0-d259556aa502","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.034904199Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:24.034916139Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035074465Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035104864Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035115804Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035210841Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035227Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"b691267b-003e-445d-a2f0-d259556aa502 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035265869Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035275619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035285419Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035295048Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035304568Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035317668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035327538Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"ds_connection connected from Ok(127.0.0.1:43939)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035398576Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"ds_connection connected from Ok(127.0.0.1:40565)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035423825Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"ds_connection connected from Ok(127.0.0.1:51756)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035435285Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.035992619Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036032438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036162794Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036279901Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.0363049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[0]R flush_numbers[0..12]: [290, 284, 290, 290, 289, 289, 288, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036506274Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[0]R generation[0..12]: [181, 175, 181, 181, 180, 180, 180, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036531184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036543163Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[1]R flush_numbers[0..12]: [290, 284, 290, 290, 289, 289, 288, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036553973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[1]R generation[0..12]: [181, 175, 181, 181, 180, 180, 180, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036564653Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036575682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[2]R flush_numbers[0..12]: [290, 284, 290, 290, 289, 289, 288, 263, 282, 280, 286, 284]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036586352Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[2]R generation[0..12]: [181, 175, 181, 181, 180, 180, 180, 160, 173, 172, 178, 175]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036597032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036608072Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036620571Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036630541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Next flush: 291","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036639931Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03664935Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"":"downstairs","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03665955Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03666842Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"0","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.03667829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"1","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036687979Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"client":"2","":"downstairs","session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036697929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036707189Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"b691267b-003e-445d-a2f0-d259556aa502 is now active with session: 785ca8fe-9dd5-4809-b3ed-8c6b939cf45e","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036717368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036726028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"session_id":"785ca8fe-9dd5-4809-b3ed-8c6b939cf45e"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036735278Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Activated sub_volume b691267b-003e-445d-a2f0-d259556aa502","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036743378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.036751578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.037071049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} 01/30 Write block 197 len 32768 data: 17 18 18 17 17 15 15 13 02/30 Read block 247 len 12288 03/30 Write block 238 len 40960 data: 14 17 14 12 12 11 11 12 16 16 04/30 Read block 371 len 24576 05/30 Read block 61 len 32768 06/30 Read block 402 len 28672 07/30 Write block 539 len 32768 data: 13 14 14 11 13 12 10 9 08/30 Read block 211 len 12288 09/30 Read block 351 len 12288 10/30 Write block 527 len 24576 data: 13 12 11 13 15 15 11/30 Write block 127 len 28672 data: 12 10 8 10 9 10 10 12/30 Read block 246 len 4096 13/30 Read block 509 len 32768 14/30 Write block 329 len 32768 data: 16 14 11 10 10 12 12 10 15/30 Read block 488 len 36864 16/30 Write block 309 len 8192 data: 17 15 17/30 Read block 49 len 4096 18/30 Read block 300 len 16384 19/30 Write block 386 len 40960 data: 16 13 15 16 14 13 17 17 16 18 20/30 Read block 124 len 12288 21/30 Write block 290 len 12288 data: 6 7 12 22/30 Read block 168 len 36864 23/30 Read block 392 len 12288 24/30 Flush 25/30 Write block 463 len 4096 data: 11 26/30 Write block 138 len 20480 data: 10 9 8 11 10 27/30 Write block 33 len 32768 data: 17 20 20 18 17 20 18 19 28/30 Read block 309 len 20480 29/30 Write block 533 len 12288 data: 13 13 11 30/30 Write block 412 len 36864 data: 10 7 8 8 9 12 11 13 18 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 10 Done Done Sent false 1003 Acked Read 6 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Write 3 Sent Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 1 29 32 Done 28 29 1 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:24.046193292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:24.046380787Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1867,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 F F F 1 020-039 175 175 175 284 284 284 T T F <--- 2 040-059 181 181 181 290 290 290 T T F <--- 3 060-079 181 181 181 290 290 290 F F F 4 080-099 180 180 180 289 289 289 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 182 182 180 291 291 288 T T F <--- 7 140-159 160 160 160 263 263 263 T T F <--- 8 160-179 173 173 173 282 282 282 F F F 9 180-199 182 182 172 291 291 280 F F T <--- 10 200-219 182 182 178 291 291 286 F F T <--- 11 220-239 182 182 175 291 291 284 F F F <--- 12 240-259 182 182 174 291 291 283 F F F <--- 13 260-279 173 173 173 282 282 282 F F F 14 280-299 182 182 180 291 291 288 F F F <--- 15 300-319 182 182 180 291 291 289 F F F <--- 16 320-339 182 182 174 291 291 283 F F F <--- 17 340-359 172 172 172 280 280 280 F F F 18 360-379 181 181 181 290 290 290 F F F 19 380-399 182 182 175 291 291 284 F F F <--- 20 400-419 180 180 180 288 288 288 T T F <--- 21 420-439 178 178 178 286 286 286 T T F <--- 22 440-459 175 175 175 284 284 284 F F F 23 460-479 179 179 179 287 287 287 T T F <--- 24 480-499 178 178 178 285 285 285 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 182 182 179 291 291 287 T T F <--- 27 540-559 182 182 180 291 291 289 F F F <--- 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 182, Max flush: 291 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 F F F 1 020-039 183 183 183 292 292 292 F F F 2 040-059 183 183 183 292 292 292 F F F 3 060-079 181 181 181 290 290 290 F F F 4 080-099 180 180 180 289 289 289 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 183 183 183 292 292 292 F F F 7 140-159 183 183 183 292 292 292 F F F 8 160-179 173 173 173 282 282 282 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 182 182 182 291 291 291 F F F 11 220-239 182 182 182 291 291 291 F F F 12 240-259 182 182 182 291 291 291 F F F 13 260-279 173 173 173 282 282 282 F F F 14 280-299 182 182 182 291 291 291 F F F 15 300-319 182 182 182 291 291 291 F F F 16 320-339 182 182 182 291 291 291 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 181 181 181 290 290 290 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 183 183 183 292 292 292 F F F 21 420-439 183 183 183 292 292 292 F F F 22 440-459 175 175 175 284 284 284 F F F 23 460-479 183 183 183 292 292 292 F F F 24 480-499 178 178 178 285 285 285 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 183, Max flush: 292 Begin loop 92 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.315673733Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:28.322868901Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.3229051Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.329873694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330161366Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330197685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330207334Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330219384Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Crucible 672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9 has session id: 843d9b37-c88b-4ff6-960b-b6658f2dc642","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330543535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Upstairs opts: Upstairs UUID: 672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330568684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Crucible stats registered with UUID: 672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330578504Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:28.330587564Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330738439Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330754809Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330765029Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330912605Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330923314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330941684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330950473Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330960253Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330970113Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330979843Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.330989912Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.331026981Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"ds_connection connected from Ok(127.0.0.1:35221)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.331048391Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"ds_connection connected from Ok(127.0.0.1:45111)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.33105962Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"ds_connection connected from Ok(127.0.0.1:52499)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.33107055Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.331705112Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.331736911Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.331852418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332003594Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332020993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[0]R flush_numbers[0..12]: [290, 292, 292, 290, 289, 289, 292, 292, 282, 291, 291, 291]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.33214464Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[0]R generation[0..12]: [181, 183, 183, 181, 180, 180, 183, 183, 173, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.33215609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332166989Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[1]R flush_numbers[0..12]: [290, 292, 292, 290, 289, 289, 292, 292, 282, 291, 291, 291]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332177479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[1]R generation[0..12]: [181, 183, 183, 181, 180, 180, 183, 183, 173, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332187929Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332198668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[2]R flush_numbers[0..12]: [290, 292, 292, 290, 289, 289, 292, 292, 282, 291, 291, 291]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332209228Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[2]R generation[0..12]: [181, 183, 183, 181, 180, 180, 183, 183, 173, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332219798Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332230678Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332240007Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332249677Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Next flush: 293","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332258997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332270376Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"":"downstairs","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332280216Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332288976Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"0","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332298736Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"1","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332308435Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"client":"2","":"downstairs","session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332318315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332327595Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9 is now active with session: 843d9b37-c88b-4ff6-960b-b6658f2dc642","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332337515Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332346194Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"session_id":"843d9b37-c88b-4ff6-960b-b6658f2dc642"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332355294Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Activated sub_volume 672cc4ae-fdb5-4c88-9839-1f0ffc0a27d9","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332363304Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332371474Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.332742763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} 01/30 Write block 62 len 36864 data: 8 9 10 14 15 14 15 13 14 02/30 Read block 392 len 20480 03/30 Read block 36 len 8192 04/30 Flush 05/30 Write block 442 len 20480 data: 12 12 13 18 16 06/30 Write block 474 len 36864 data: 15 14 16 12 10 9 9 7 12 07/30 Read block 227 len 40960 08/30 Write block 52 len 28672 data: 14 18 17 13 16 17 15 09/30 Read block 187 len 4096 10/30 Write block 266 len 16384 data: 13 16 17 19 11/30 Read block 138 len 28672 12/30 Write block 151 len 32768 data: 18 17 16 10 12 12 9 9 13/30 Read block 249 len 20480 14/30 Write block 367 len 4096 data: 20 15/30 Write block 405 len 8192 data: 14 12 16/30 Write block 437 len 40960 data: 12 10 8 9 10 13 13 14 19 17 17/30 Write block 416 len 24576 data: 10 13 12 14 19 17 18/30 Read block 335 len 20480 19/30 Read block 263 len 36864 20/30 Read block 438 len 4096 21/30 Write block 452 len 32768 data: 13 14 12 13 16 16 16 12 22/30 Write block 207 len 32768 data: 13 19 19 18 16 17 16 14 23/30 Write block 407 len 20480 data: 13 13 11 11 11 24/30 Write block 304 len 36864 data: 16 16 19 20 16 18 16 16 18 25/30 Write block 227 len 20480 data: 13 15 15 15 15 26/30 Write block 170 len 36864 data: 9 9 12 14 11 14 14 14 12 27/30 Write block 227 len 36864 data: 14 16 16 16 16 11 13 15 14 28/30 Read block 321 len 8192 29/30 Read block 499 len 12288 30/30 Write block 318 len 24576 data: 15 15 14 15 15 18 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Sent Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:28.341199885Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:28.3414069Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1875,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 F F F 1 020-039 183 183 183 292 292 292 F F F 2 040-059 183 183 183 292 292 292 F T T <--- 3 060-079 181 184 184 290 293 293 F F F <--- 4 080-099 180 180 180 289 289 289 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 183 183 183 292 292 292 F F F 7 140-159 183 183 183 292 292 292 F T T <--- 8 160-179 173 173 173 282 282 282 F T T <--- 9 180-199 182 182 182 291 291 291 F F F 10 200-219 182 182 182 291 291 291 F T T <--- 11 220-239 182 182 182 291 291 291 F T T <--- 12 240-259 182 182 182 291 291 291 F F F 13 260-279 173 173 173 282 282 282 F T T <--- 14 280-299 182 182 182 291 291 291 F F F 15 300-319 182 182 182 291 291 291 F T T <--- 16 320-339 182 182 182 291 291 291 F T T <--- 17 340-359 172 172 172 280 280 280 F F F 18 360-379 181 181 181 290 290 290 F T T <--- 19 380-399 182 182 182 291 291 291 F F F 20 400-419 183 183 183 292 292 292 F T T <--- 21 420-439 183 183 183 292 292 292 F T T <--- 22 440-459 175 175 175 284 284 284 F T T <--- 23 460-479 183 183 183 292 292 292 F T T <--- 24 480-499 178 178 178 285 285 285 F T T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 184, Max flush: 293 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 F F F 1 020-039 183 183 183 292 292 292 F F F 2 040-059 185 185 185 294 294 294 F F F 3 060-079 184 184 184 293 293 293 F F F 4 080-099 180 180 180 289 289 289 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 183 183 183 292 292 292 F F F 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 185 185 185 294 294 294 F F F 11 220-239 185 185 185 294 294 294 F F F 12 240-259 182 182 182 291 291 291 F F F 13 260-279 185 185 185 294 294 294 F F F 14 280-299 182 182 182 291 291 291 F F F 15 300-319 185 185 185 294 294 294 F F F 16 320-339 185 185 185 294 294 294 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 185 185 185 294 294 294 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 185 185 185 294 294 294 F F F 21 420-439 185 185 185 294 294 294 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 185 185 185 294 294 294 F F F 24 480-499 185 185 185 294 294 294 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 178 178 178 285 285 285 F F F Max gen: 185, Max flush: 294 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.616866682Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625216438Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625505959Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625535609Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625546488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625558318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Crucible a307b4b7-992b-44bd-84df-ce16afdf2d6b has session id: b596c6f3-dbbb-4add-b5b4-4209d44a50bc","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62585705Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Upstairs opts: Upstairs UUID: a307b4b7-992b-44bd-84df-ce16afdf2d6b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625886419Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Crucible stats registered with UUID: a307b4b7-992b-44bd-84df-ce16afdf2d6b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.625897978Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:32.625907368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626049374Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626080023Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626090793Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62620357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626221839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"a307b4b7-992b-44bd-84df-ce16afdf2d6b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626249829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626259368Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626271588Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626281608Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626291317Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626301257Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626311347Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:59681)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626355496Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:62181)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626366675Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:35780)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.626410054Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627026777Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627053346Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627151343Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627303309Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627318418Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[0]R flush_numbers[0..12]: [290, 292, 294, 293, 289, 289, 292, 294, 294, 291, 294, 294]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627403086Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[0]R generation[0..12]: [181, 183, 185, 184, 180, 180, 183, 185, 185, 182, 185, 185]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627414676Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627428345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[1]R flush_numbers[0..12]: [290, 292, 294, 293, 289, 289, 292, 294, 294, 291, 294, 294]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627439005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[1]R generation[0..12]: [181, 183, 185, 184, 180, 180, 183, 185, 185, 182, 185, 185]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627449655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627460694Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[2]R flush_numbers[0..12]: [290, 292, 294, 293, 289, 289, 292, 294, 294, 291, 294, 294]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627471434Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[2]R generation[0..12]: [181, 183, 185, 184, 180, 180, 183, 185, 185, 182, 185, 185]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627481994Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627492974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627502413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627512223Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Next flush: 295","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627521433Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627530673Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"":"downstairs","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627540412Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627549162Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"0","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627558892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"1","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627568541Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"client":"2","":"downstairs","session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627579921Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627589201Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"a307b4b7-992b-44bd-84df-ce16afdf2d6b is now active with session: b596c6f3-dbbb-4add-b5b4-4209d44a50bc","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627599111Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62760844Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62761655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"session_id":"b596c6f3-dbbb-4add-b5b4-4209d44a50bc"} {"msg":"Activated sub_volume a307b4b7-992b-44bd-84df-ce16afdf2d6b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62762603Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.62763415Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.627956871Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} 01/30 Flush 02/30 Write block 45 len 8192 data: 22 24 03/30 Read block 564 len 40960 04/30 Write block 487 len 40960 data: 12 12 13 16 16 15 11 11 14 13 05/30 Read block 139 len 12288 06/30 Read block 137 len 16384 07/30 Flush 08/30 Write block 124 len 20480 data: 11 13 12 13 11 09/30 Flush 10/30 Flush 11/30 Write block 76 len 4096 data: 9 12/30 Write block 584 len 36864 data: 17 15 15 15 17 14 13 12 11 13/30 Read block 503 len 8192 14/30 Flush 15/30 Write block 432 len 24576 data: 11 12 12 14 13 13 16/30 Write block 401 len 8192 data: 15 18 17/30 Write block 84 len 40960 data: 14 14 13 15 14 13 11 9 10 10 18/30 Read block 7 len 36864 19/30 Read block 284 len 8192 20/30 Read block 321 len 28672 21/30 Write block 584 len 36864 data: 18 16 16 16 18 15 14 13 12 22/30 Read block 358 len 4096 23/30 Write block 236 len 40960 data: 12 11 15 18 15 13 13 12 12 13 24/30 Write block 21 len 20480 data: 13 16 16 18 18 25/30 Write block 6 len 40960 data: 13 17 16 15 14 14 15 15 17 16 26/30 Read block 518 len 36864 27/30 Read block 456 len 16384 28/30 Read block 497 len 36864 29/30 Read block 514 len 40960 30/30 Read block 146 len 12288 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 4 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 25 26 Done 29 28 4 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:32.636784862Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:32.636965817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1883,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 290 290 290 T T F <--- 1 020-039 183 183 183 292 292 292 T T F <--- 2 040-059 186 186 185 296 296 294 F F T <--- 3 060-079 186 186 184 299 299 293 F F F <--- 4 080-099 180 180 180 289 289 289 T T F <--- 5 100-119 180 180 180 289 289 289 F F F 6 120-139 186 186 183 297 297 292 F F F <--- 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 185 185 185 294 294 294 F F F 11 220-239 185 185 185 294 294 294 T T F <--- 12 240-259 182 182 182 291 291 291 T T F <--- 13 260-279 185 185 185 294 294 294 F F F 14 280-299 182 182 182 291 291 291 F F F 15 300-319 185 185 185 294 294 294 F F F 16 320-339 185 185 185 294 294 294 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 185 185 185 294 294 294 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 185 185 185 294 294 294 T T F <--- 21 420-439 185 185 185 294 294 294 T T F <--- 22 440-459 185 185 185 294 294 294 F F F 23 460-479 185 185 185 294 294 294 F F F 24 480-499 186 186 185 296 296 294 F F T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 186 186 178 299 299 285 T T F <--- Max gen: 186, Max flush: 299 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 300 300 300 F F F 1 020-039 187 187 187 300 300 300 F F F 2 040-059 186 186 186 296 296 296 F F F 3 060-079 186 186 186 299 299 299 F F F 4 080-099 187 187 187 300 300 300 F F F 5 100-119 180 180 180 289 289 289 F F F 6 120-139 186 186 186 297 297 297 F F F 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 185 185 185 294 294 294 F F F 11 220-239 187 187 187 300 300 300 F F F 12 240-259 187 187 187 300 300 300 F F F 13 260-279 185 185 185 294 294 294 F F F 14 280-299 182 182 182 291 291 291 F F F 15 300-319 185 185 185 294 294 294 F F F 16 320-339 185 185 185 294 294 294 F F F 17 340-359 172 172 172 280 280 280 F F F 18 360-379 185 185 185 294 294 294 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 187 187 187 300 300 300 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 185 185 185 294 294 294 F F F 24 480-499 186 186 186 296 296 296 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 178 178 178 285 285 285 F F F 29 580-599 187 187 187 300 300 300 F F F Max gen: 187, Max flush: 300 Begin loop 94 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.894694308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.902920637Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903217488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903243028Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903251667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903263327Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Crucible b5f647b4-7c31-4caa-9315-3ea01c05ade0 has session id: fcdd0aa7-644b-418b-a8bc-ad924532216c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903617757Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Upstairs opts: Upstairs UUID: b5f647b4-7c31-4caa-9315-3ea01c05ade0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903651956Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Crucible stats registered with UUID: b5f647b4-7c31-4caa-9315-3ea01c05ade0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903663256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:36.903672535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903817971Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903835831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903846071Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903982447Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.903992536Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"b5f647b4-7c31-4caa-9315-3ea01c05ade0 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904036355Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904046005Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904056015Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904065984Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904075604Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904085914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904096104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"ds_connection connected from Ok(127.0.0.1:62411)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904138472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"ds_connection connected from Ok(127.0.0.1:36627)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904166652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"ds_connection connected from Ok(127.0.0.1:49443)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.90423906Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904759255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904783944Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.904872792Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905087866Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905123195Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[0]R flush_numbers[0..12]: [300, 300, 296, 299, 300, 289, 297, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905177273Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[0]R generation[0..12]: [187, 187, 186, 186, 187, 180, 186, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905188523Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905199643Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[1]R flush_numbers[0..12]: [300, 300, 296, 299, 300, 289, 297, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905210592Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[1]R generation[0..12]: [187, 187, 186, 186, 187, 180, 186, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905221472Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905232462Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[2]R flush_numbers[0..12]: [300, 300, 296, 299, 300, 289, 297, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905243061Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[2]R generation[0..12]: [187, 187, 186, 186, 187, 180, 186, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905256431Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905267531Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.90527709Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.90528722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Next flush: 301","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.90529667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.90530613Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"":"downstairs","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905315819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905324639Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"0","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905334319Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"1","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905343878Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"client":"2","":"downstairs","session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905353668Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905362718Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"b5f647b4-7c31-4caa-9315-3ea01c05ade0 is now active with session: fcdd0aa7-644b-418b-a8bc-ad924532216c","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905372708Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905381407Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"session_id":"fcdd0aa7-644b-418b-a8bc-ad924532216c"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905390527Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Activated sub_volume b5f647b4-7c31-4caa-9315-3ea01c05ade0","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905398587Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905408167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.905780906Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} 01/30 Write block 107 len 12288 data: 14 16 16 02/30 Read block 550 len 12288 03/30 Read block 270 len 32768 04/30 Flush 05/30 Write block 306 len 4096 data: 20 06/30 Read block 170 len 16384 07/30 Flush 08/30 Write block 100 len 8192 data: 12 14 09/30 Read block 325 len 16384 10/30 Read block 208 len 8192 11/30 Write block 564 len 4096 data: 16 12/30 Write block 585 len 36864 data: 17 17 17 19 16 15 14 13 11 13/30 Write block 8 len 40960 data: 17 16 15 15 16 16 18 17 13 11 14/30 Write block 272 len 28672 data: 14 15 14 14 13 15 15 15/30 Write block 127 len 24576 data: 14 12 9 11 10 11 16/30 Flush 17/30 Write block 477 len 32768 data: 13 11 10 10 8 13 12 12 18/30 Flush 19/30 Write block 285 len 36864 data: 6 7 8 7 6 7 8 13 14 20/30 Write block 332 len 40960 data: 11 11 13 13 11 10 9 12 14 14 21/30 Read block 240 len 32768 22/30 Write block 130 len 12288 data: 12 11 12 23/30 Write block 420 len 4096 data: 20 24/30 Flush 25/30 Write block 308 len 24576 data: 17 19 17 17 19 16 26/30 Write block 357 len 32768 data: 10 14 16 16 16 13 13 14 27/30 Read block 446 len 12288 28/30 Read block 325 len 32768 29/30 Write block 530 len 4096 data: 14 30/30 Read block 72 len 8192 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 8 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:36.914283357Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:36.914460772Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1891,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 187 303 303 300 F F F <--- 1 020-039 187 187 187 300 300 300 F F F 2 040-059 186 186 186 296 296 296 F F F 3 060-079 186 186 186 299 299 299 F F F 4 080-099 187 187 187 300 300 300 F F F 5 100-119 188 188 180 303 303 289 F F T <--- 6 120-139 188 188 186 305 305 297 F F F <--- 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 185 185 185 294 294 294 F F F 11 220-239 187 187 187 300 300 300 F F F 12 240-259 187 187 187 300 300 300 F F F 13 260-279 188 188 185 303 303 294 F F F <--- 14 280-299 188 188 182 305 305 291 F F F <--- 15 300-319 188 188 185 302 302 294 T T F <--- 16 320-339 188 188 185 305 305 294 F F F <--- 17 340-359 188 188 172 305 305 280 T T F <--- 18 360-379 185 185 185 294 294 294 T T F <--- 19 380-399 182 182 182 291 291 291 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 187 305 305 300 F F F <--- 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 185 304 304 294 F F F <--- 24 480-499 188 188 186 304 304 296 F F F <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 183 183 183 292 292 292 T T F <--- 27 540-559 182 182 182 291 291 291 F F F 28 560-579 188 188 178 303 303 285 F F F <--- 29 580-599 188 188 187 303 303 300 F F F <--- Max gen: 188, Max flush: 305 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 187 187 187 300 300 300 F F F 2 040-059 186 186 186 296 296 296 F F F 3 060-079 186 186 186 299 299 299 F F F 4 080-099 187 187 187 300 300 300 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 F F F 9 180-199 182 182 182 291 291 291 F F F 10 200-219 185 185 185 294 294 294 F F F 11 220-239 187 187 187 300 300 300 F F F 12 240-259 187 187 187 300 300 300 F F F 13 260-279 188 188 188 303 303 303 F F F 14 280-299 188 188 188 305 305 305 F F F 15 300-319 189 189 189 306 306 306 F F F 16 320-339 188 188 188 305 305 305 F F F 17 340-359 189 189 189 306 306 306 F F F 18 360-379 189 189 189 306 306 306 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 188 304 304 304 F F F 24 480-499 188 188 188 304 304 304 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 182 182 182 291 291 291 F F F 28 560-579 188 188 188 303 303 303 F F F 29 580-599 188 188 188 303 303 303 F F F Max gen: 189, Max flush: 306 Begin loop 95 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.18226481Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.190483359Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.190779621Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19080387Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19081283Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.190824699Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Crucible 33945d05-4a7d-4b5d-89b8-0409defecfec has session id: 7ddb03bc-0bac-4d2e-872b-9651b14693d3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19115711Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Upstairs opts: Upstairs UUID: 33945d05-4a7d-4b5d-89b8-0409defecfec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191182789Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Crucible stats registered with UUID: 33945d05-4a7d-4b5d-89b8-0409defecfec","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191192969Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:41.191240068Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191407063Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191436652Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191449742Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191581248Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191594578Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"33945d05-4a7d-4b5d-89b8-0409defecfec active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191633247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191642836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191652556Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191662236Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191671786Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191681685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191691685Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:54686)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191732954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:60727)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191758993Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47945)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.191835741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192445084Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192488103Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192531511Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192760995Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192784404Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[0]R flush_numbers[0..12]: [303, 300, 296, 299, 300, 303, 305, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192845353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[0]R generation[0..12]: [188, 187, 186, 186, 187, 188, 188, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192857332Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192868452Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[1]R flush_numbers[0..12]: [303, 300, 296, 299, 300, 303, 305, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192879082Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[1]R generation[0..12]: [188, 187, 186, 186, 187, 188, 188, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192889741Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192900661Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[2]R flush_numbers[0..12]: [303, 300, 296, 299, 300, 303, 305, 294, 294, 291, 294, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192911411Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[2]R generation[0..12]: [188, 187, 186, 186, 187, 188, 188, 185, 185, 182, 185, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192921961Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19293292Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19294264Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.19295244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Next flush: 307","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192961839Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192973779Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"":"downstairs","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192983589Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.192992379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"0","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193002118Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"1","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193011778Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"client":"2","":"downstairs","session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193021498Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193030817Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"33945d05-4a7d-4b5d-89b8-0409defecfec is now active with session: 7ddb03bc-0bac-4d2e-872b-9651b14693d3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193040837Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193049507Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"session_id":"7ddb03bc-0bac-4d2e-872b-9651b14693d3"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193058567Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Activated sub_volume 33945d05-4a7d-4b5d-89b8-0409defecfec","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193066856Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193075206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.193439126Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} 01/30 Write block 254 len 12288 data: 15 12 12 02/30 Write block 591 len 36864 data: 15 14 12 11 9 7 7 7 6 03/30 Read block 35 len 12288 04/30 Read block 533 len 12288 05/30 Write block 304 len 8192 data: 17 17 06/30 Write block 32 len 4096 data: 18 07/30 Read block 327 len 28672 08/30 Write block 350 len 24576 data: 11 14 17 16 12 11 09/30 Flush 10/30 Write block 565 len 36864 data: 13 10 10 10 8 9 8 7 9 11/30 Read block 23 len 20480 12/30 Flush 13/30 Write block 542 len 24576 data: 12 14 13 11 10 12 14/30 Read block 574 len 8192 15/30 Read block 372 len 12288 16/30 Read block 183 len 16384 17/30 Read block 236 len 12288 18/30 Read block 584 len 20480 19/30 Write block 161 len 8192 data: 13 15 20/30 Read block 233 len 8192 21/30 Write block 257 len 20480 data: 10 10 11 9 9 22/30 Write block 197 len 16384 data: 18 19 19 18 23/30 Read block 235 len 40960 24/30 Write block 64 len 8192 data: 11 15 25/30 Read block 188 len 32768 26/30 Read block 429 len 36864 27/30 Write block 553 len 24576 data: 19 15 16 16 17 15 28/30 Read block 82 len 36864 29/30 Read block 590 len 36864 30/30 Read block 525 len 36864 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 28 29 Done 29 30 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:41.201980726Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:41.202171181Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1899,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 187 307 307 300 F F F <--- 2 040-059 186 186 186 296 296 296 F F F 3 060-079 186 186 186 299 299 299 T T F <--- 4 080-099 187 187 187 300 300 300 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 185 185 185 294 294 294 F F F 8 160-179 185 185 185 294 294 294 T T F <--- 9 180-199 182 182 182 291 291 291 T T F <--- 10 200-219 185 185 185 294 294 294 T T F <--- 11 220-239 187 187 187 300 300 300 F F F 12 240-259 190 190 187 307 307 300 T T T <--- 13 260-279 188 188 188 303 303 303 T T F <--- 14 280-299 188 188 188 305 305 305 F F F 15 300-319 190 190 189 307 307 306 F F F <--- 16 320-339 188 188 188 305 305 305 F F F 17 340-359 190 190 189 307 307 306 F F F <--- 18 360-379 189 189 189 306 306 306 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 188 304 304 304 F F F 24 480-499 188 188 188 304 304 304 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 182 182 182 291 291 291 T T F <--- 28 560-579 190 190 188 308 308 303 F F F <--- 29 580-599 190 190 188 307 307 303 F F T <--- Max gen: 190, Max flush: 308 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 190 307 307 307 F F F 2 040-059 186 186 186 296 296 296 F F F 3 060-079 191 191 191 309 309 309 F F F 4 080-099 187 187 187 300 300 300 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 185 185 185 294 294 294 F F F 8 160-179 191 191 191 309 309 309 F F F 9 180-199 191 191 191 309 309 309 F F F 10 200-219 191 191 191 309 309 309 F F F 11 220-239 187 187 187 300 300 300 F F F 12 240-259 191 191 191 309 309 309 F F F 13 260-279 191 191 191 309 309 309 F F F 14 280-299 188 188 188 305 305 305 F F F 15 300-319 190 190 190 307 307 307 F F F 16 320-339 188 188 188 305 305 305 F F F 17 340-359 190 190 190 307 307 307 F F F 18 360-379 189 189 189 306 306 306 F F F 19 380-399 182 182 182 291 291 291 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 188 304 304 304 F F F 24 480-499 188 188 188 304 304 304 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 191 191 191 309 309 309 F F F 28 560-579 190 190 190 308 308 308 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 191, Max flush: 309 Begin loop 96 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.462587696Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:45.469711396Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.469770684Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.476544884Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.476802247Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.476827526Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.476835716Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.476847305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Crucible 055d4092-1bf9-43e4-b463-c7372d16ff81 has session id: e581d14d-71dd-4657-a2c6-097867b3d8e3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477162336Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Upstairs opts: Upstairs UUID: 055d4092-1bf9-43e4-b463-c7372d16ff81, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477186196Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Crucible stats registered with UUID: 055d4092-1bf9-43e4-b463-c7372d16ff81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477196115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:45.477211255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477361241Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.47739221Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.47740342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477533036Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477546356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"055d4092-1bf9-43e4-b463-c7372d16ff81 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477580615Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477592074Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477601954Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477611674Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477621154Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477631003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477640873Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47085)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477708171Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:41790)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.47773385Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:51597)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.477821378Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478366443Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478388572Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478509819Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478698903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478708863Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[0]R flush_numbers[0..12]: [303, 307, 296, 309, 300, 303, 305, 294, 309, 309, 309, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478845209Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[0]R generation[0..12]: [188, 190, 186, 191, 187, 188, 188, 185, 191, 191, 191, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478859089Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478870318Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[1]R flush_numbers[0..12]: [303, 307, 296, 309, 300, 303, 305, 294, 309, 309, 309, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478881128Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[1]R generation[0..12]: [188, 190, 186, 191, 187, 188, 188, 185, 191, 191, 191, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478891838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478902748Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[2]R flush_numbers[0..12]: [303, 307, 296, 309, 300, 303, 305, 294, 309, 309, 309, 300]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478913467Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[2]R generation[0..12]: [188, 190, 186, 191, 187, 188, 188, 185, 191, 191, 191, 187]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478924207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478935137Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478944606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478954706Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Next flush: 310","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478964246Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478973606Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"":"downstairs","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478983365Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.478992255Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"0","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479002155Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"1","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479013314Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"client":"2","":"downstairs","session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479023184Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479032244Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"055d4092-1bf9-43e4-b463-c7372d16ff81 is now active with session: e581d14d-71dd-4657-a2c6-097867b3d8e3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479042444Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479051233Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"session_id":"e581d14d-71dd-4657-a2c6-097867b3d8e3"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479060433Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Activated sub_volume 055d4092-1bf9-43e4-b463-c7372d16ff81","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479068243Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479076373Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.479379174Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} 01/30 Write block 50 len 24576 data: 18 15 15 19 18 14 02/30 Read block 553 len 8192 03/30 Read block 174 len 16384 04/30 Write block 375 len 4096 data: 16 05/30 Read block 527 len 32768 06/30 Flush 07/30 Write block 344 len 8192 data: 13 13 08/30 Read block 335 len 28672 09/30 Write block 82 len 8192 data: 14 13 10/30 Write block 195 len 36864 data: 18 17 19 20 20 19 18 16 16 11/30 Flush 12/30 Write block 333 len 32768 data: 12 14 14 12 11 10 13 15 13/30 Write block 378 len 12288 data: 14 15 15 14/30 Write block 480 len 28672 data: 11 9 14 13 13 11 10 15/30 Read block 356 len 24576 16/30 Read block 245 len 36864 17/30 Write block 241 len 40960 data: 14 14 13 13 14 17 17 16 20 23 18/30 Read block 386 len 28672 19/30 Write block 43 len 28672 data: 25 23 23 25 24 21 19 20/30 Write block 239 len 8192 data: 19 16 21/30 Write block 90 len 12288 data: 12 10 11 22/30 Write block 219 len 40960 data: 13 15 16 13 13 12 15 16 15 17 23/30 Write block 562 len 24576 data: 15 15 17 14 11 11 24/30 Read block 86 len 20480 25/30 Write block 159 len 20480 data: 8 11 14 16 15 26/30 Read block 128 len 12288 27/30 Read block 261 len 40960 28/30 Read block 336 len 8192 29/30 Write block 296 len 20480 data: 10 11 14 13 16 30/30 Read block 166 len 16384 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 8 Done Done Done false 1012 Acked Write 3 Done Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 0 0 17 Done 2 19 19 40 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:45.487805307Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:45.488001582Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1907,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 190 307 307 307 F F F 2 040-059 192 192 192 310 310 310 F T T <--- 3 060-079 191 191 191 309 309 309 F F F 4 080-099 192 192 192 311 311 311 F T T <--- 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 185 185 185 294 294 294 F T T <--- 8 160-179 191 191 191 309 309 309 F T T <--- 9 180-199 192 192 192 311 311 311 F F F 10 200-219 192 192 192 311 311 311 F T T <--- 11 220-239 187 187 187 300 300 300 F T T <--- 12 240-259 191 191 191 309 309 309 F T T <--- 13 260-279 191 191 191 309 309 309 F F F 14 280-299 188 188 188 305 305 305 F T T <--- 15 300-319 190 190 190 307 307 307 F T T <--- 16 320-339 188 188 188 305 305 305 T T T 17 340-359 192 192 192 311 311 311 T T T 18 360-379 192 192 192 310 310 310 T T T 19 380-399 182 182 182 291 291 291 T T T 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 188 304 304 304 F F F 24 480-499 188 188 188 304 304 304 F T T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 191 191 191 309 309 309 F F F 28 560-579 190 190 190 308 308 308 F T T <--- 29 580-599 190 190 190 307 307 307 F F F Max gen: 192, Max flush: 311 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 190 307 307 307 F F F 2 040-059 193 193 193 312 312 312 F F F 3 060-079 191 191 191 309 309 309 F F F 4 080-099 193 193 193 312 312 312 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 193 193 193 312 312 312 F F F 8 160-179 193 193 193 312 312 312 F F F 9 180-199 192 192 192 311 311 311 F F F 10 200-219 193 193 193 312 312 312 F F F 11 220-239 193 193 193 312 312 312 F F F 12 240-259 193 193 193 312 312 312 F F F 13 260-279 191 191 191 309 309 309 F F F 14 280-299 193 193 193 312 312 312 F F F 15 300-319 193 193 193 312 312 312 F F F 16 320-339 193 193 193 312 312 312 F F F 17 340-359 193 193 193 312 312 312 F F F 18 360-379 193 193 193 312 312 312 F F F 19 380-399 193 193 193 312 312 312 F F F 20 400-419 187 187 187 300 300 300 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 185 185 185 294 294 294 F F F 23 460-479 188 188 188 304 304 304 F F F 24 480-499 193 193 193 312 312 312 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 191 191 191 309 309 309 F F F 28 560-579 193 193 193 312 312 312 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 193, Max flush: 312 Begin loop 97 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.761800951Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.769988251Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770273493Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770302602Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770311192Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770328311Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Crucible eb069891-02da-4748-996c-08bc1aa5104b has session id: 03303c80-2d34-4e50-8513-8e42cd35b889","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770663712Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Upstairs opts: Upstairs UUID: eb069891-02da-4748-996c-08bc1aa5104b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770692881Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Crucible stats registered with UUID: eb069891-02da-4748-996c-08bc1aa5104b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770705831Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:49.770715161Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770862256Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770894026Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.770905345Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771023152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771033012Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"eb069891-02da-4748-996c-08bc1aa5104b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771050771Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771059611Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771069421Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.77107899Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.77108875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.77109886Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771108759Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"ds_connection connected from Ok(127.0.0.1:53966)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771121739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"ds_connection connected from Ok(127.0.0.1:34809)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771187207Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"ds_connection connected from Ok(127.0.0.1:55049)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771212827Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.77181287Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771832739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.771855479Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772123091Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.77214583Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[0]R flush_numbers[0..12]: [303, 307, 312, 309, 312, 303, 305, 312, 312, 311, 312, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772192359Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[0]R generation[0..12]: [188, 190, 193, 191, 193, 188, 188, 193, 193, 192, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772204299Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772215528Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[1]R flush_numbers[0..12]: [303, 307, 312, 309, 312, 303, 305, 312, 312, 311, 312, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772226308Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[1]R generation[0..12]: [188, 190, 193, 191, 193, 188, 188, 193, 193, 192, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772236938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772247907Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[2]R flush_numbers[0..12]: [303, 307, 312, 309, 312, 303, 305, 312, 312, 311, 312, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772258547Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[2]R generation[0..12]: [188, 190, 193, 191, 193, 188, 188, 193, 193, 192, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772271977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772282996Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772292476Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772302386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Next flush: 313","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772311646Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772321095Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"":"downstairs","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772330865Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772339655Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"0","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772349305Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"1","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772359004Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"client":"2","":"downstairs","session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772368914Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772378134Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"eb069891-02da-4748-996c-08bc1aa5104b is now active with session: 03303c80-2d34-4e50-8513-8e42cd35b889","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772388104Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772396853Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"session_id":"03303c80-2d34-4e50-8513-8e42cd35b889"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772405903Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Activated sub_volume eb069891-02da-4748-996c-08bc1aa5104b","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772414093Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772422323Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.772759003Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} 01/30 Write block 453 len 36864 data: 15 13 14 17 17 17 13 11 6 02/30 Write block 212 len 16384 data: 18 17 15 14 03/30 Read block 487 len 28672 04/30 Write block 359 len 40960 data: 17 17 17 14 14 15 18 21 21 20 05/30 Write block 177 len 20480 data: 15 13 14 13 17 06/30 Write block 451 len 12288 data: 14 14 16 07/30 Write block 343 len 32768 data: 13 14 14 13 13 13 11 12 08/30 Flush 09/30 Write block 81 len 24576 data: 14 15 14 15 15 14 10/30 Write block 344 len 40960 data: 15 15 14 14 14 12 13 15 18 17 11/30 Write block 362 len 28672 data: 15 15 16 19 22 22 21 12/30 Write block 156 len 28672 data: 13 10 10 9 12 15 17 13/30 Write block 551 len 20480 data: 18 20 20 16 17 14/30 Write block 212 len 12288 data: 19 18 16 15/30 Write block 445 len 40960 data: 20 18 15 14 12 8 15 15 17 14 16/30 Read block 140 len 4096 17/30 Flush 18/30 Write block 407 len 28672 data: 14 14 12 12 12 11 8 19/30 Read block 169 len 40960 20/30 Read block 109 len 20480 21/30 Write block 160 len 4096 data: 13 22/30 Write block 445 len 36864 data: 21 19 16 15 13 9 16 16 18 23/30 Flush 24/30 Write block 492 len 32768 data: 16 12 12 15 14 12 10 7 25/30 Read block 8 len 28672 26/30 Read block 426 len 24576 27/30 Read block 399 len 32768 28/30 Write block 58 len 36864 data: 16 17 14 10 9 10 12 16 16 29/30 Write block 410 len 12288 data: 13 13 12 30/30 Read block 218 len 20480 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 4 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 9 Done Sent Done false 1028 Acked Write 3 Done Sent Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 2 31 Done 30 1 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:49.781142037Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:49.781334142Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1915,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 190 307 307 307 F F F 2 040-059 193 193 193 312 312 312 T F T <--- 3 060-079 191 191 191 309 309 309 T F T <--- 4 080-099 194 193 194 314 312 314 F F F <--- 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 194 193 194 314 312 314 F F F <--- 8 160-179 194 193 194 315 312 315 F F F <--- 9 180-199 194 192 194 313 311 313 F F F <--- 10 200-219 194 193 194 314 312 314 F F F <--- 11 220-239 193 193 193 312 312 312 F F F 12 240-259 193 193 193 312 312 312 F F F 13 260-279 191 191 191 309 309 309 F F F 14 280-299 193 193 193 312 312 312 F F F 15 300-319 193 193 193 312 312 312 F F F 16 320-339 193 193 193 312 312 312 F F F 17 340-359 194 193 194 314 312 314 F F F <--- 18 360-379 194 193 194 314 312 314 F F F <--- 19 380-399 193 193 193 312 312 312 F F F 20 400-419 194 187 194 315 300 315 T F T <--- 21 420-439 188 188 188 305 305 305 F F F 22 440-459 194 185 194 315 294 315 F T F <--- 23 460-479 194 188 194 313 304 313 F T F <--- 24 480-499 193 193 193 312 312 312 T F T <--- 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 194 191 194 314 309 314 F F F <--- 28 560-579 193 193 193 312 312 312 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 194, Max flush: 315 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 303 303 303 F F F 1 020-039 190 190 190 307 307 307 F F F 2 040-059 195 195 195 316 316 316 F F F 3 060-079 195 195 195 316 316 316 F F F 4 080-099 194 194 194 314 314 314 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 194 194 194 314 314 314 F F F 8 160-179 194 194 194 315 315 315 F F F 9 180-199 194 194 194 313 313 313 F F F 10 200-219 194 194 194 314 314 314 F F F 11 220-239 193 193 193 312 312 312 F F F 12 240-259 193 193 193 312 312 312 F F F 13 260-279 191 191 191 309 309 309 F F F 14 280-299 193 193 193 312 312 312 F F F 15 300-319 193 193 193 312 312 312 F F F 16 320-339 193 193 193 312 312 312 F F F 17 340-359 194 194 194 314 314 314 F F F 18 360-379 194 194 194 314 314 314 F F F 19 380-399 193 193 193 312 312 312 F F F 20 400-419 195 195 195 316 316 316 F F F 21 420-439 188 188 188 305 305 305 F F F 22 440-459 194 194 194 315 315 315 F F F 23 460-479 194 194 194 313 313 313 F F F 24 480-499 195 195 195 316 316 316 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 189 189 306 306 306 F F F 27 540-559 194 194 194 314 314 314 F F F 28 560-579 193 193 193 312 312 312 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 195, Max flush: 316 Begin loop 98 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.043327353Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:54.050448892Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.050501441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.057376518Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.0576436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.057673379Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.057682149Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.057694229Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Crucible 26765529-86d4-4464-b8b0-bd4e76a783b8 has session id: f0008a6a-e664-436c-a501-8b84835f4d85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058030409Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Upstairs opts: Upstairs UUID: 26765529-86d4-4464-b8b0-bd4e76a783b8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058071488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Crucible stats registered with UUID: 26765529-86d4-4464-b8b0-bd4e76a783b8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058082738Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:54.058094538Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058249563Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058280032Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058291152Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058410469Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058424478Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"26765529-86d4-4464-b8b0-bd4e76a783b8 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058442758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058451548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058461107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058470667Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058480157Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058490056Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058499836Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:43603)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058552115Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:52289)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058577364Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"ds_connection connected from Ok(127.0.0.1:38311)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.058587904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059279064Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059303974Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059315463Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059607325Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059631814Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[0]R flush_numbers[0..12]: [303, 307, 316, 316, 314, 303, 305, 314, 315, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059692763Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[0]R generation[0..12]: [188, 190, 195, 195, 194, 188, 188, 194, 194, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059704702Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059716212Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[1]R flush_numbers[0..12]: [303, 307, 316, 316, 314, 303, 305, 314, 315, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059726992Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[1]R generation[0..12]: [188, 190, 195, 195, 194, 188, 188, 194, 194, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059737811Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059748851Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[2]R flush_numbers[0..12]: [303, 307, 316, 316, 314, 303, 305, 314, 315, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059759641Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[2]R generation[0..12]: [188, 190, 195, 195, 194, 188, 188, 194, 194, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.05977049Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.0597818Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.05979441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.05980441Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059813909Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059823499Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"":"downstairs","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059833679Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059842548Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"0","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059852658Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"1","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059862488Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"client":"2","":"downstairs","session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059872738Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059881997Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"26765529-86d4-4464-b8b0-bd4e76a783b8 is now active with session: f0008a6a-e664-436c-a501-8b84835f4d85","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059892077Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059900957Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"session_id":"f0008a6a-e664-436c-a501-8b84835f4d85"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059910167Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Activated sub_volume 26765529-86d4-4464-b8b0-bd4e76a783b8","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059918206Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.059926436Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.060285176Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} 01/30 Write block 263 len 36864 data: 10 13 15 14 17 18 20 21 16 02/30 Write block 150 len 28672 data: 16 19 18 17 11 13 14 03/30 Write block 152 len 36864 data: 19 18 12 14 15 11 11 10 14 04/30 Write block 350 len 36864 data: 14 16 19 18 13 12 12 11 15 05/30 Write block 38 len 4096 data: 21 06/30 Write block 2 len 40960 data: 8 8 12 10 14 18 18 17 16 16 07/30 Read block 467 len 8192 08/30 Write block 58 len 32768 data: 17 18 15 11 10 11 13 17 09/30 Read block 436 len 36864 10/30 Read block 91 len 16384 11/30 Write block 534 len 8192 data: 14 12 12/30 Read block 218 len 40960 13/30 Read block 226 len 12288 14/30 Write block 248 len 28672 data: 17 21 24 21 18 17 16 15/30 Write block 439 len 28672 data: 9 10 11 14 14 15 22 16/30 Write block 561 len 20480 data: 15 16 16 18 15 17/30 Read block 335 len 36864 18/30 Write block 416 len 28672 data: 11 14 13 15 21 18 18 19/30 Write block 392 len 4096 data: 18 20/30 Read block 168 len 24576 21/30 Read block 556 len 20480 22/30 Read block 329 len 20480 23/30 Read block 288 len 24576 24/30 Flush 25/30 Flush 26/30 Write block 156 len 8192 data: 16 12 27/30 Read block 8 len 24576 28/30 Write block 460 len 36864 data: 12 7 11 12 10 11 9 10 10 29/30 Write block 15 len 4096 data: 18 30/30 Write block 285 len 36864 data: 7 8 9 8 7 8 9 14 15 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 7 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 3 35 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:54.068719539Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:54.068915373Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1923,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 196 196 303 317 317 F T T <--- 1 020-039 190 196 196 307 317 317 F F F <--- 2 040-059 195 196 196 316 317 317 F F F <--- 3 060-079 195 196 196 316 317 317 F F F <--- 4 080-099 194 194 194 314 314 314 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 194 196 196 314 317 317 F T T <--- 8 160-179 194 196 196 315 317 317 F F F <--- 9 180-199 194 194 194 313 313 313 F F F 10 200-219 194 194 194 314 314 314 F F F 11 220-239 193 193 193 312 312 312 F F F 12 240-259 193 196 196 312 317 317 F F F <--- 13 260-279 191 196 196 309 317 317 T F F <--- 14 280-299 193 193 193 312 312 312 F T T <--- 15 300-319 193 193 193 312 312 312 F F F 16 320-339 193 193 193 312 312 312 F F F 17 340-359 194 196 196 314 317 317 F F F <--- 18 360-379 194 194 194 314 314 314 F F F 19 380-399 193 196 196 312 317 317 F F F <--- 20 400-419 195 196 196 316 317 317 F F F <--- 21 420-439 188 196 196 305 317 317 F F F <--- 22 440-459 194 196 196 315 317 317 F F F <--- 23 460-479 194 194 194 313 313 313 F T T <--- 24 480-499 195 195 195 316 316 316 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 189 196 196 306 317 317 F F F <--- 27 540-559 194 194 194 314 314 314 F F F 28 560-579 193 196 196 312 317 317 F F F <--- 29 580-599 190 190 190 307 307 307 F F F Max gen: 196, Max flush: 317 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 197 197 197 318 318 318 F F F 1 020-039 196 196 196 317 317 317 F F F 2 040-059 196 196 196 317 317 317 F F F 3 060-079 196 196 196 317 317 317 F F F 4 080-099 194 194 194 314 314 314 F F F 5 100-119 188 188 188 303 303 303 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 197 197 197 318 318 318 F F F 8 160-179 196 196 196 317 317 317 F F F 9 180-199 194 194 194 313 313 313 F F F 10 200-219 194 194 194 314 314 314 F F F 11 220-239 193 193 193 312 312 312 F F F 12 240-259 196 196 196 317 317 317 F F F 13 260-279 196 196 196 317 317 317 F F F 14 280-299 197 197 197 318 318 318 F F F 15 300-319 193 193 193 312 312 312 F F F 16 320-339 193 193 193 312 312 312 F F F 17 340-359 196 196 196 317 317 317 F F F 18 360-379 194 194 194 314 314 314 F F F 19 380-399 196 196 196 317 317 317 F F F 20 400-419 196 196 196 317 317 317 F F F 21 420-439 196 196 196 317 317 317 F F F 22 440-459 196 196 196 317 317 317 F F F 23 460-479 197 197 197 318 318 318 F F F 24 480-499 195 195 195 316 316 316 F F F 25 500-519 168 168 168 274 274 274 F F F 26 520-539 196 196 196 317 317 317 F F F 27 540-559 194 194 194 314 314 314 F F F 28 560-579 196 196 196 317 317 317 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 197, Max flush: 318 Begin loop 99 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.341160107Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:58.348319875Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.348375624Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355169533Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355439535Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e4a11e41192a288088e06a0dccb1dfefc86d6532\",\n git_commit_timestamp: \"2026-09-24T21:53:41.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355463574Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355471904Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355483614Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Crucible 7a77ddf4-7f1b-4421-978d-06faa5883eed has session id: 4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355821934Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Upstairs opts: Upstairs UUID: 7a77ddf4-7f1b-4421-978d-06faa5883eed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355852413Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Crucible stats registered with UUID: 7a77ddf4-7f1b-4421-978d-06faa5883eed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.355863453Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:58.355872623Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356002169Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356032938Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356045838Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356197024Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356212013Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"7a77ddf4-7f1b-4421-978d-06faa5883eed active request set","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356227883Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356236973Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356246962Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356256762Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356266342Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356276352Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:36548)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35631647Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:33468)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35632758Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.3563375Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63885)","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356415058Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.356975722Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357000841Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357012731Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357278723Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357288543Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[0]R flush_numbers[0..12]: [318, 317, 317, 317, 314, 303, 305, 318, 317, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357359581Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[0]R generation[0..12]: [197, 196, 196, 196, 194, 188, 188, 197, 196, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357371011Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35738208Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[1]R flush_numbers[0..12]: [318, 317, 317, 317, 314, 303, 305, 318, 317, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35739315Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[1]R generation[0..12]: [197, 196, 196, 196, 194, 188, 188, 197, 196, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35740386Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.35741463Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[2]R flush_numbers[0..12]: [318, 317, 317, 317, 314, 303, 305, 318, 317, 313, 314, 312]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357425219Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[2]R generation[0..12]: [197, 196, 196, 196, 194, 188, 188, 197, 196, 194, 194, 193]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357435829Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357446739Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357456348Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357466148Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Next flush: 319","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357475428Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357486638Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"":"downstairs","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357496337Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357504977Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"0","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357514627Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"1","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357524406Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"client":"2","":"downstairs","session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357534286Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357543356Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"7a77ddf4-7f1b-4421-978d-06faa5883eed is now active with session: 4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357553266Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357561985Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"session_id":"4d8ebc15-fa07-41b7-a7c0-9547e4cc6a6d"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357571025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Activated sub_volume 7a77ddf4-7f1b-4421-978d-06faa5883eed","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357579025Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357587045Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.357951584Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} 01/30 Write block 100 len 40960 data: 13 15 14 11 12 12 12 15 17 17 02/30 Write block 265 len 12288 data: 16 15 18 03/30 Read block 45 len 8192 04/30 Write block 272 len 24576 data: 15 16 15 15 14 16 05/30 Read block 214 len 36864 06/30 Write block 236 len 40960 data: 13 12 16 20 17 15 15 14 14 15 07/30 Write block 342 len 16384 data: 11 14 16 16 08/30 Write block 518 len 4096 data: 17 09/30 Write block 163 len 32768 data: 16 13 13 11 12 14 9 10 10/30 Read block 312 len 4096 11/30 Read block 258 len 40960 12/30 Read block 263 len 20480 13/30 Flush 14/30 Write block 393 len 40960 data: 18 17 19 19 18 17 14 15 16 19 15/30 Write block 561 len 28672 data: 16 17 17 19 16 12 12 16/30 Read block 408 len 28672 17/30 Read block 570 len 16384 18/30 Read block 213 len 24576 19/30 Write block 320 len 4096 data: 15 20/30 Read block 113 len 8192 21/30 Write block 438 len 12288 data: 11 10 11 22/30 Write block 47 len 16384 data: 25 22 20 19 23/30 Write block 202 len 28672 data: 17 17 14 14 12 14 20 24/30 Write block 379 len 20480 data: 16 16 17 16 15 25/30 Write block 293 len 16384 data: 16 10 11 11 26/30 Read block 24 len 40960 27/30 Write block 177 len 4096 data: 16 28/30 Read block 479 len 28672 29/30 Write block 330 len 12288 data: 15 12 12 30/30 Write block 302 len 28672 data: 18 15 18 18 21 21 18 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Write 4 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Write 8 Done Done Done false 1009 Acked Read 1 Done Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 2 2 24 Done 10 28 28 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-24T22:15:58.365769405Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-24T22:15:58.36593349Z","hostname":"w-01M3AQB297KF2486S65ZN6DMCS","pid":1931,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 197 197 197 318 318 318 F F F 1 020-039 196 196 196 317 317 317 F F F 2 040-059 196 196 196 317 317 317 F T T <--- 3 060-079 196 196 196 317 317 317 F F F 4 080-099 194 194 194 314 314 314 F F F 5 100-119 188 198 198 303 319 319 T F F <--- 6 120-139 188 188 188 305 305 305 F F F 7 140-159 197 197 197 318 318 318 F F F 8 160-179 196 198 198 317 319 319 T T T <--- 9 180-199 194 194 194 313 313 313 F F F 10 200-219 194 194 194 314 314 314 F T T <--- 11 220-239 193 198 198 312 319 319 T F F <--- 12 240-259 196 198 198 317 319 319 T F F <--- 13 260-279 196 198 198 317 319 319 T F F <--- 14 280-299 197 197 197 318 318 318 F T T <--- 15 300-319 193 193 193 312 312 312 F T T <--- 16 320-339 193 193 193 312 312 312 F T T <--- 17 340-359 196 198 198 317 319 319 T F F <--- 18 360-379 194 194 194 314 314 314 F T T <--- 19 380-399 196 196 196 317 317 317 F T T <--- 20 400-419 196 196 196 317 317 317 F T T <--- 21 420-439 196 196 196 317 317 317 F T T <--- 22 440-459 196 196 196 317 317 317 F T T <--- 23 460-479 197 197 197 318 318 318 F F F 24 480-499 195 195 195 316 316 316 F F F 25 500-519 168 198 198 274 319 319 T F F <--- 26 520-539 196 196 196 317 317 317 F F F 27 540-559 194 194 194 314 314 314 F F F 28 560-579 196 196 196 317 317 317 F T T <--- 29 580-599 190 190 190 307 307 307 F F F Max gen: 198, Max flush: 319 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 197 197 197 318 318 318 F F F 1 020-039 196 196 196 317 317 317 F F F 2 040-059 199 199 199 320 320 320 F F F 3 060-079 196 196 196 317 317 317 F F F 4 080-099 194 194 194 314 314 314 F F F 5 100-119 198 198 198 319 319 319 F F F 6 120-139 188 188 188 305 305 305 F F F 7 140-159 197 197 197 318 318 318 F F F 8 160-179 199 199 199 320 320 320 F F F 9 180-199 194 194 194 313 313 313 F F F 10 200-219 199 199 199 320 320 320 F F F 11 220-239 198 198 198 319 319 319 F F F 12 240-259 198 198 198 319 319 319 F F F 13 260-279 198 198 198 319 319 319 F F F 14 280-299 199 199 199 320 320 320 F F F 15 300-319 199 199 199 320 320 320 F F F 16 320-339 199 199 199 320 320 320 F F F 17 340-359 198 198 198 319 319 319 F F F 18 360-379 199 199 199 320 320 320 F F F 19 380-399 199 199 199 320 320 320 F F F 20 400-419 199 199 199 320 320 320 F F F 21 420-439 199 199 199 320 320 320 F F F 22 440-459 199 199 199 320 320 320 F F F 23 460-479 197 197 197 318 318 318 F F F 24 480-499 195 195 195 316 316 316 F F F 25 500-519 198 198 198 319 319 319 F F F 26 520-539 196 196 196 317 317 317 F F F 27 540-559 194 194 194 314 314 314 F F F 28 560-579 199 199 199 320 320 320 F F F 29 580-599 190 190 190 307 307 307 F F F Max gen: 199, Max flush: 320 7:28 Test duration Test completed real 7:28.479988299 user 25.954843477 sys 14.545199287 trap 0.019983188 tflt 0.066174590 dflt 0.069481095 kflt 0.003441166 lock 11:55:00.318680578 slp 36:05.685181597 lat 2.358456862 stop 0.020430942