Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.38250795 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.037994076 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.041053064 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1193 1194 1195 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.823141301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.856371295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.858458063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.858724339Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.858852301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.858892871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"Crucible ed623d2a-1c6f-4b4f-994c-ce9b620fed7b has session id: 0c0ad2c8-f989-4657-8966-50b7347430b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.860347332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Upstairs opts: Upstairs UUID: ed623d2a-1c6f-4b4f-994c-ce9b620fed7b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.860609569Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Crucible stats registered with UUID: ed623d2a-1c6f-4b4f-994c-ce9b620fed7b","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.860645119Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:58:59.860669878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.862524899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.862720575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.862785643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.86327578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.863571765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"ed623d2a-1c6f-4b4f-994c-ce9b620fed7b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.863672193Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.863712923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.863752421Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.863789985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.864075008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.864129315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.864169514Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:41830)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.864912791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61166)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.864982388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:45548)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.86500859Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.869654368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.869714657Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.869962205Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871030083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871126763Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871264285Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871305225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871448568Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871496092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871528356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871563134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.87159642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871628123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871664535Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.87169256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871721447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871747969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871788409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"":"downstairs","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871820933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871847045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"0","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871886292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"1","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871934117Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"client":"2","":"downstairs","session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.871976009Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.872005547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"ed623d2a-1c6f-4b4f-994c-ce9b620fed7b is now active with session: 0c0ad2c8-f989-4657-8966-50b7347430b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.872038943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.872067349Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.872090665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198,"session_id":"0c0ad2c8-f989-4657-8966-50b7347430b3"} {"msg":"Activated sub_volume ed623d2a-1c6f-4b4f-994c-ce9b620fed7b","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:58:59.872347312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:03.993983916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1198} Begin loop 1 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.03968973Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:04.054912831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.054984422Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.071806398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.072616076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.072703618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.072729379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.072764759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"Crucible 436eaed0-29b4-4dd3-9ad2-a2feed307437 has session id: 226f2f1a-d6d5-410d-be32-23d3bfc46d5f","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.073689874Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Upstairs opts: Upstairs UUID: 436eaed0-29b4-4dd3-9ad2-a2feed307437, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.073757809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Crucible stats registered with UUID: 436eaed0-29b4-4dd3-9ad2-a2feed307437","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.073789651Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:04.073816304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.074448001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.074507197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.07454898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075165927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075211637Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"436eaed0-29b4-4dd3-9ad2-a2feed307437 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075413646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075464997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075505006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075543122Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075579103Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075616877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075655253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60345)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.075971948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:37865)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.076059501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55763)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.076090191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.078085038Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.078387825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.078640043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.078944303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079066925Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079264054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079294274Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079319233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079342189Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079364282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.07938828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079410473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079432667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079456685Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079476494Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079497395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079517244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079537494Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"":"downstairs","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079563385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079582643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"0","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079603634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"1","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079624405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"client":"2","":"downstairs","session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079645758Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079692119Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"436eaed0-29b4-4dd3-9ad2-a2feed307437 is now active with session: 226f2f1a-d6d5-410d-be32-23d3bfc46d5f","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079714814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079733902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200,"session_id":"226f2f1a-d6d5-410d-be32-23d3bfc46d5f"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079753991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} {"msg":"Activated sub_volume 436eaed0-29b4-4dd3-9ad2-a2feed307437","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:04.079771556Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 239 len 28672 data: 2 2 2 2 2 2 2 02/30 Write block 378 len 8192 data: 2 2 03/30 Read block 58 len 4096 04/30 Read block 168 len 4096 05/30 Read block 543 len 16384 06/30 Write block 325 len 16384 data: 2 2 2 2 07/30 Write block 115 len 16384 data: 2 2 2 2 08/30 Read block 6 len 28672 09/30 Write block 85 len 36864 data: 2 2 2 2 2 2 2 2 2 10/30 Read block 37 len 8192 11/30 Write block 504 len 32768 data: 2 2 2 2 2 2 2 2 12/30 Read block 294 len 12288 13/30 Write block 442 len 36864 data: 2 2 2 2 2 2 2 2 2 14/30 Read block 511 len 4096 15/30 Read block 442 len 12288 16/30 Read block 9 len 12288 17/30 Read block 407 len 8192 18/30 Read block 99 len 16384 19/30 Write block 137 len 16384 data: 2 2 2 2 20/30 Read block 32 len 4096 21/30 Flush 22/30 Write block 562 len 36864 data: 2 2 2 2 2 2 2 2 2 23/30 Write block 76 len 32768 data: 2 2 2 2 2 2 2 2 24/30 Write block 151 len 28672 data: 2 2 2 2 2 2 2 25/30 Write block 414 len 12288 data: 2 2 2 26/30 Write block 428 len 36864 data: 2 2 2 2 2 2 2 2 2 27/30 Read block 522 len 36864 28/30 Write block 533 len 4096 data: 2 29/30 Read block 442 len 16384 30/30 Write block 489 len 24576 data: 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:04.109596171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1200} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F T T <--- 4 080-099 1 2 2 1 2 2 F T T <--- 5 100-119 1 2 2 1 2 2 F F F <--- 6 120-139 1 2 2 1 2 2 F F F <--- 7 140-159 1 2 2 1 2 2 F T T <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 2 2 1 2 2 T F F <--- 12 240-259 1 2 2 1 2 2 T F F <--- 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 2 2 1 2 2 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 2 2 1 2 2 T F F <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 1 1 1 1 1 1 F T T <--- 21 420-439 1 1 1 1 1 1 F T T <--- 22 440-459 1 2 2 1 2 2 F F F <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F T T <--- 25 500-519 1 2 2 1 2 2 F F F <--- 26 520-539 1 1 1 1 1 1 F T T <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F T T <--- 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 2 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 2 2 2 2 2 2 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 3 3 3 3 3 3 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 2 2 2 2 2 2 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 2 2 2 2 2 2 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 3 3 3 3 3 3 F F F 22 440-459 2 2 2 2 2 2 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 3 3 3 3 3 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 3 Begin loop 2 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.104428166Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.129789393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.130317655Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.130406571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.130435799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.130476048Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"Crucible 6d58748f-d42b-41b3-b361-3186bb2ec5a4 has session id: 4dd7c07b-00c2-476b-bcb5-597732b888f9","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.131940147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Upstairs opts: Upstairs UUID: 6d58748f-d42b-41b3-b361-3186bb2ec5a4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.13210863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Crucible stats registered with UUID: 6d58748f-d42b-41b3-b361-3186bb2ec5a4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.132147176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:09.132179129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.132587274Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.132647273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.132679426Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133111298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133148562Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"6d58748f-d42b-41b3-b361-3186bb2ec5a4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133243399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133277466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133308938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133350791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133396692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133452742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.13351731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:38054)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133577989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:61799)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133633689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43348)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.133939613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.135856836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.1360715Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.136483343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.13695815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137172473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137364982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137406164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137443698Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137476963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137509448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137545759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.1376019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 3, 3, 2, 2, 3, 1, 1, 1, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.13765258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T05:59:09.137711236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.13775398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137784771Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Next flush: 4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137814169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137843687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"":"downstairs","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137874458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137902303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"0","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137933494Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"1","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137964095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"client":"2","":"downstairs","session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.137995547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.138024965Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"6d58748f-d42b-41b3-b361-3186bb2ec5a4 is now active with session: 4dd7c07b-00c2-476b-bcb5-597732b888f9","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.138066186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.138094462Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208,"session_id":"4dd7c07b-00c2-476b-bcb5-597732b888f9"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.138150653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} {"msg":"Activated sub_volume 6d58748f-d42b-41b3-b361-3186bb2ec5a4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:09.138194379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 278 len 16384 data: 2 2 2 2 02/30 Read block 102 len 32768 03/30 Read block 215 len 28672 04/30 Read block 233 len 28672 05/30 Read block 452 len 24576 06/30 Read block 4 len 24576 07/30 Flush 08/30 Write block 419 len 24576 data: 2 2 2 2 2 2 09/30 Read block 49 len 28672 10/30 Write block 525 len 16384 data: 2 2 2 2 11/30 Write block 359 len 28672 data: 2 2 2 2 2 2 2 12/30 Read block 92 len 40960 13/30 Write block 202 len 24576 data: 2 2 2 2 2 2 14/30 Read block 418 len 16384 15/30 Write block 558 len 8192 data: 2 2 16/30 Write block 81 len 40960 data: 3 3 3 2 3 3 3 3 3 3 17/30 Read block 433 len 32768 18/30 Read block 214 len 36864 19/30 Read block 315 len 16384 20/30 Read block 494 len 24576 21/30 Read block 498 len 8192 22/30 Write block 97 len 32768 data: 2 2 2 2 2 2 2 2 23/30 Read block 531 len 12288 24/30 Read block 146 len 36864 25/30 Read block 183 len 40960 26/30 Write block 328 len 20480 data: 3 2 2 2 2 27/30 Read block 61 len 32768 28/30 Read block 283 len 24576 29/30 Write block 258 len 8192 data: 2 2 30/30 Write block 283 len 40960 data: 2 2 2 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Read 8 Done Done Sent false 1002 Acked Read 7 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1006)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:09.176470582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1208} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 3 3 3 3 3 3 T T F <--- 5 100-119 2 2 2 2 2 2 T T F <--- 6 120-139 2 2 2 2 2 2 F F F 7 140-159 3 3 3 3 3 3 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 T T F <--- 11 220-239 2 2 2 2 2 2 F F F 12 240-259 2 2 2 2 2 2 T T F <--- 13 260-279 4 4 1 4 4 1 F F F <--- 14 280-299 4 4 1 4 4 1 T T F <--- 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 T T F <--- 17 340-359 1 1 1 1 1 1 T T F <--- 18 360-379 2 2 2 2 2 2 T T F <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 3 3 3 3 3 3 T T F <--- 21 420-439 3 3 3 3 3 3 T T F <--- 22 440-459 2 2 2 2 2 2 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 3 3 3 3 3 T T F <--- 27 540-559 1 1 1 1 1 1 T T F <--- 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 4, Max flush: 4 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 3 3 3 3 3 3 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 5 5 5 5 5 5 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 3 3 3 3 3 3 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 5 5 5 5 5 5 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 5 5 5 5 5 5 F F F 13 260-279 4 4 4 4 4 4 F F F 14 280-299 5 5 5 5 5 5 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 5 5 5 5 5 5 F F F 18 360-379 5 5 5 5 5 5 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 5 5 5 5 5 5 F F F 21 420-439 5 5 5 5 5 5 F F F 22 440-459 2 2 2 2 2 2 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 5 5 5 5 5 5 F F F 27 540-559 5 5 5 5 5 5 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 5, Max flush: 5 Begin loop 3 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.898032085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.923704135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.924550044Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.924613109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.92463406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.92466431Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"Crucible 19cc3920-2246-495e-8a19-a9f02ed6a864 has session id: 653d51c6-c6ef-4333-ad2a-d6910faba350","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.926146685Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Upstairs opts: Upstairs UUID: 19cc3920-2246-495e-8a19-a9f02ed6a864, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.926248766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Crucible stats registered with UUID: 19cc3920-2246-495e-8a19-a9f02ed6a864","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.926278405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:13.926304045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.926638125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.9266965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.926730387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927038755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927082041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"19cc3920-2246-495e-8a19-a9f02ed6a864 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927203601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927232918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927259311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927284791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927312976Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927422833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"ds_connection connected from Ok(127.0.0.1:39286)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.927530997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.92758843Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"ds_connection connected from Ok(127.0.0.1:36701)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.92761868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"ds_connection connected from Ok(127.0.0.1:44451)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.928163224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.92986872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.93007154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.930134785Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.930660031Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931042055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931101493Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931142183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931203975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931245036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931283261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931344733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931400372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 3, 5, 5, 2, 3, 1, 1, 5, 2]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931451503ZWait for a query_work_queue command to finish before sending IO ","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931511502Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931541942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931589335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931638192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.93166793Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"":"downstairs","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.93170321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931747618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"0","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T05:59:13.931817004Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"1","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931866242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"client":"2","":"downstairs","session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931918525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.931963864Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"19cc3920-2246-495e-8a19-a9f02ed6a864 is now active with session: 653d51c6-c6ef-4333-ad2a-d6910faba350","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.932016568Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.932063832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216,"session_id":"653d51c6-c6ef-4333-ad2a-d6910faba350"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.932093179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} {"msg":"Activated sub_volume 19cc3920-2246-495e-8a19-a9f02ed6a864","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:13.932123469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 398 len 36864 data: 2 2 2 2 2 2 2 2 2 02/30 Read block 77 len 28672 03/30 Read block 537 len 20480 04/30 Flush 05/30 Write block 519 len 28672 data: 2 2 2 2 2 2 3 06/30 Read block 129 len 20480 07/30 Write block 531 len 32768 data: 2 2 3 2 2 2 2 2 08/30 Read block 32 len 20480 09/30 Read block 537 len 12288 10/30 Read block 390 len 20480 11/30 Flush 12/30 Write block 29 len 16384 data: 2 2 2 2 13/30 Flush 14/30 Write block 226 len 28672 data: 2 2 2 2 2 2 2 15/30 Read block 20 len 8192 16/30 Flush 17/30 Write block 222 len 28672 data: 2 2 2 2 3 3 3 18/30 Flush 19/30 Write block 37 len 20480 data: 2 2 2 2 2 20/30 Write block 146 len 24576 data: 2 2 2 2 2 3 21/30 Read block 431 len 32768 22/30 Flush 23/30 Write block 17 len 20480 data: 2 2 2 2 2 24/30 Write block 68 len 4096 data: 2 25/30 Write block 194 len 16384 data: 2 2 2 2 26/30 Read block 373 len 32768 27/30 Write block 399 len 36864 data: 3 3 3 3 3 3 3 3 2 28/30 Read block 219 len 32768 29/30 Write block 10 len 16384 data: 2 2 2 2 30/30 Write block 254 len 28672 data: 2 2 2 2 3 3 2 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 26 2 30 Done 24 0 24 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1003)) Some(JobId(1021)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:13.963340662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1216} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 T F T <--- 1 020-039 6 1 6 11 1 11 T F T <--- 2 040-059 6 1 6 11 1 11 F F F <--- 3 060-079 3 3 3 3 3 3 T F T <--- 4 080-099 5 5 5 5 5 5 F F F 5 100-119 5 5 5 5 5 5 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 6 3 6 11 3 11 F F F <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 T F T <--- 10 200-219 5 5 5 5 5 5 F F F 11 220-239 6 2 6 10 2 10 F F F <--- 12 240-259 5 5 5 5 5 5 T F T <--- 13 260-279 4 4 4 4 4 4 T F T <--- 14 280-299 5 5 5 5 5 5 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 5 5 5 5 5 5 F F F 18 360-379 5 5 5 5 5 5 F F F 19 380-399 6 6 6 6 6 6 T F T <--- 20 400-419 6 6 6 6 6 6 T F T <--- 21 420-439 5 5 5 5 5 5 F F F 22 440-459 2 2 2 2 2 2 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 6 2 6 7 2 7 F F F <--- 26 520-539 6 5 6 7 5 7 F F F <--- 27 540-559 5 5 5 5 5 5 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 6, Max flush: 11 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 12 12 12 F F F 1 020-039 7 7 7 12 12 12 F F F 2 040-059 6 6 6 11 11 11 F F F 3 060-079 7 7 7 12 12 12 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 5 5 5 5 5 5 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 6 6 6 11 11 11 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F F F 10 200-219 5 5 5 5 5 5 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 7 7 12 12 12 F F F 13 260-279 7 7 7 12 12 12 F F F 14 280-299 5 5 5 5 5 5 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 5 5 5 5 5 5 F F F 18 360-379 5 5 5 5 5 5 F F F 19 380-399 7 7 7 12 12 12 F F F 20 400-419 7 7 7 12 12 12 F F F 21 420-439 5 5 5 5 5 5 F F F 22 440-459 2 2 2 2 2 2 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 6 6 6 7 7 7 F F F 26 520-539 6 6 6 7 7 7 F F F 27 540-559 5 5 5 5 5 5 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 7, Max flush: 12 Begin loop 4 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.639657321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.66423927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.664785929Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.664977467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.665002276Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.665042705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"Crucible 5bad8fec-5693-4ca9-955c-76422259332e has session id: 4a8bd4c6-2e62-4857-88b2-13a62b65c9b2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666398651Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Upstairs opts: Upstairs UUID: 5bad8fec-5693-4ca9-955c-76422259332e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666459181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Crucible stats registered with UUID: 5bad8fec-5693-4ca9-955c-76422259332e","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666500131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:18.666525532Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666841756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666892155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.666920471Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667244801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667279109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"5bad8fec-5693-4ca9-955c-76422259332e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667523351Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667553781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667580523Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667606264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667631855Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667657996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667858352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:36221)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.667991815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:46408)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.668235175Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:41886)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.668296066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.66993022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.669985149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.670322534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.670900836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.670935414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[0]R flush_numbers[0..12]: [12, 12, 11, 12, 5, 5, 2, 11, 1, 12, 5, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671081382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[0]R generation[0..12]: [7, 7, 6, 7, 5, 5, 2, 6, 1, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.6711207Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671157322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[1]R flush_numbers[0..12]: [12, 12, 11, 12, 5, 5, 2, 11, 1, 12, 5, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671191028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[1]R generation[0..12]: [7, 7, 6, 7, 5, 5, 2, 6, 1, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671223773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671259794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[2]R flush_numbers[0..12]: [12, 12, 11, 12, 5, 5, 2, 11, 1, 12, 5, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.67129297Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[2]R generation[0..12]: [7, 7, 6, 7, 5, 5, 2, 6, 1, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671325313Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671361194Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671401403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671432365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Next flush: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671461131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671490529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"":"downstairs","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.67152102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671548885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"0","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671579214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"1","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.67162243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"client":"2","":"downstairs","session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"All required reconciliation work is completed","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-03-06T05:59:18.671676306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671723409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"5bad8fec-5693-4ca9-955c-76422259332e is now active with session: 4a8bd4c6-2e62-4857-88b2-13a62b65c9b2","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671760823Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671788608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224,"session_id":"4a8bd4c6-2e62-4857-88b2-13a62b65c9b2"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671817625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} {"msg":"Activated sub_volume 5bad8fec-5693-4ca9-955c-76422259332e","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:18.671843145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 271 len 8192 data: 2 2 02/30 Write block 110 len 16384 data: 2 2 2 2 03/30 Write block 379 len 40960 data: 3 2 2 2 2 2 2 2 2 2 04/30 Read block 188 len 40960 05/30 Write block 42 len 12288 data: 2 2 2 06/30 Flush 07/30 Write block 445 len 24576 data: 3 3 3 3 3 3 08/30 Read block 393 len 12288 09/30 Write block 460 len 16384 data: 2 2 2 2 10/30 Flush 11/30 Write block 520 len 32768 data: 3 3 3 3 3 4 3 3 12/30 Read block 403 len 20480 13/30 Read block 279 len 32768 14/30 Read block 1 len 32768 15/30 Flush 16/30 Flush 17/30 Write block 267 len 4096 data: 2 18/30 Write block 50 len 8192 data: 2 2 19/30 Read block 578 len 8192 20/30 Write block 34 len 36864 data: 2 2 2 3 3 3 3 3 3 21/30 Read block 424 len 4096 22/30 Read block 163 len 24576 23/30 Write block 430 len 32768 data: 3 3 3 3 3 3 3 2 24/30 Read block 21 len 32768 25/30 Read block 379 len 8192 26/30 Write block 200 len 28672 data: 2 2 3 3 3 3 3 27/30 Read block 137 len 12288 28/30 Read block 494 len 40960 29/30 Write block 350 len 28672 data: 2 2 2 2 2 2 2 30/30 Write block 115 len 8192 data: 3 3 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Write 4 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 10 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Read 3 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:18.7032741Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1224} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 12 12 12 F F F 1 020-039 7 7 7 12 12 12 T F T <--- 2 040-059 8 6 8 13 11 13 T F T <--- 3 060-079 7 7 7 12 12 12 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 8 5 8 13 5 13 T F T <--- 6 120-139 2 2 2 2 2 2 F F F 7 140-159 6 6 6 11 11 11 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F F F 10 200-219 5 5 5 5 5 5 T F T <--- 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 7 7 12 12 12 F F F 13 260-279 8 7 8 13 12 13 T F T <--- 14 280-299 5 5 5 5 5 5 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 5 5 5 5 5 5 T F T <--- 18 360-379 8 5 8 13 5 13 F F F <--- 19 380-399 8 7 8 13 12 13 F F F <--- 20 400-419 7 7 7 12 12 12 F F F 21 420-439 5 5 5 5 5 5 T F T <--- 22 440-459 8 2 8 14 2 14 F F F <--- 23 460-479 8 1 8 14 1 14 F F F <--- 24 480-499 3 3 3 3 3 3 F F F 25 500-519 6 6 6 7 7 7 F F F 26 520-539 8 6 8 15 7 15 F F F <--- 27 540-559 5 5 5 5 5 5 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 8, Max flush: 15 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 12 12 12 F F F 1 020-039 9 9 9 16 16 16 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 7 7 7 12 12 12 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 6 6 6 11 11 11 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 7 7 12 12 12 F F F 13 260-279 9 9 9 16 16 16 F F F 14 280-299 5 5 5 5 5 5 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 8 8 8 13 13 13 F F F 19 380-399 8 8 8 13 13 13 F F F 20 400-419 7 7 7 12 12 12 F F F 21 420-439 9 9 9 16 16 16 F F F 22 440-459 8 8 8 14 14 14 F F F 23 460-479 8 8 8 14 14 14 F F F 24 480-499 3 3 3 3 3 3 F F F 25 500-519 6 6 6 7 7 7 F F F 26 520-539 8 8 8 15 15 15 F F F 27 540-559 5 5 5 5 5 5 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 9, Max flush: 16 Begin loop 5 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.347907404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.374441967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.375226345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.375335059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.375377162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.375417742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"Crucible c730a770-ec20-40fd-820e-e70f6f5fb9ef has session id: f56798af-4b38-4e29-baa8-1063cf1c5257","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.376763047Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Upstairs opts: Upstairs UUID: c730a770-ec20-40fd-820e-e70f6f5fb9ef, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.376834438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Crucible stats registered with UUID: c730a770-ec20-40fd-820e-e70f6f5fb9ef","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.376867253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:23.376895749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.377383381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.377487547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.377523107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.377953085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.377992844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"c730a770-ec20-40fd-820e-e70f6f5fb9ef active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378279119Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378312294Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378343616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378388805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378419897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378451309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378538871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"ds_connection connected from Ok(127.0.0.1:51136)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378632887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"ds_connection connected from Ok(127.0.0.1:46720)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.378823944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:23.379093015Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:23.37919206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:23.379233252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.379517192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.379680264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.38075282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.380878368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.3817765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:23.381869624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.381428849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"ds_connection connected from Ok(127.0.0.1:54608)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.381640577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.384290917Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[0]R flush_numbers[0..12]: [12, 16, 16, 12, 5, 16, 2, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385354606Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[0]R generation[0..12]: [7, 9, 9, 7, 5, 9, 2, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385426157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385484883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[1]R flush_numbers[0..12]: [12, 16, 16, 12, 5, 16, 2, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385521726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[1]R generation[0..12]: [7, 9, 9, 7, 5, 9, 2, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385559751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385602595Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[2]R flush_numbers[0..12]: [12, 16, 16, 12, 5, 16, 2, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385653586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[2]R generation[0..12]: [7, 9, 9, 7, 5, 9, 2, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385806087Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385888399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385920072Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.385951514Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Next flush: 17","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386003216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386042854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"":"downstairs","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.38608639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T05:59:33.386119235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"0","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386158793Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"1","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386189824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"client":"2","":"downstairs","session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386221136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386250253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"c730a770-ec20-40fd-820e-e70f6f5fb9ef is now active with session: f56798af-4b38-4e29-baa8-1063cf1c5257","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386283439Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.386311033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232,"session_id":"f56798af-4b38-4e29-baa8-1063cf1c5257"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.38634006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} {"msg":"Activated sub_volume c730a770-ec20-40fd-820e-e70f6f5fb9ef","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:33.38636524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 373 len 12288 data: 2 2 2 02/30 Read block 530 len 8192 03/30 Read block 468 len 8192 04/30 Write block 420 len 24576 data: 3 3 3 3 3 2 05/30 Flush 06/30 Write block 559 len 40960 data: 3 2 2 3 3 3 3 3 3 3 07/30 Write block 481 len 16384 data: 2 2 2 2 08/30 Write block 579 len 12288 data: 2 2 2 09/30 Write block 547 len 24576 data: 2 2 2 2 2 2 10/30 Read block 407 len 16384 11/30 Write block 579 len 36864 data: 3 3 3 2 2 2 2 2 2 12/30 Read block 38 len 32768 13/30 Write block 596 len 8192 data: 2 2 14/30 Read block 495 len 12288 15/30 Read block 476 len 12288 16/30 Write block 528 len 40960 data: 3 2 2 3 3 4 3 3 3 3 17/30 Write block 532 len 8192 data: 4 5 18/30 Flush 19/30 Flush 20/30 Write block 402 len 8192 data: 4 4 21/30 Read block 353 len 32768 22/30 Read block 551 len 28672 23/30 Write block 120 len 40960 data: 2 2 2 2 2 2 2 2 2 2 24/30 Read block 120 len 24576 25/30 Write block 297 len 28672 data: 2 2 2 2 2 2 2 26/30 Write block 570 len 4096 data: 3 27/30 Write block 11 len 20480 data: 3 3 3 2 2 28/30 Write block 269 len 20480 data: 2 2 3 3 2 29/30 Write block 275 len 12288 data: 2 2 2 30/30 Read block 445 len 24576 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 2 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:33.421126216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1232} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 12 12 12 T T F <--- 1 020-039 9 9 9 16 16 16 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 7 7 7 12 12 12 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 2 2 2 2 2 2 T T F <--- 7 140-159 6 6 6 11 11 11 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 7 7 12 12 12 F F F 13 260-279 9 9 9 16 16 16 T T F <--- 14 280-299 5 5 5 5 5 5 T T F <--- 15 300-319 1 1 1 1 1 1 T T F <--- 16 320-339 5 5 5 5 5 5 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 10 10 8 17 17 13 F F T <--- 19 380-399 8 8 8 13 13 13 F F F 20 400-419 7 7 7 12 12 12 T T F <--- 21 420-439 10 10 9 17 17 16 F F F <--- 22 440-459 8 8 8 14 14 14 F F F 23 460-479 8 8 8 14 14 14 F F F 24 480-499 10 10 3 18 18 3 F F F <--- 25 500-519 6 6 6 7 7 7 F F F 26 520-539 10 10 8 18 18 15 F F F <--- 27 540-559 10 10 5 18 18 5 F F F <--- 28 560-579 10 10 3 18 18 3 T T F <--- 29 580-599 10 10 1 18 18 1 F F F <--- Max gen: 10, Max flush: 18 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 11 11 11 19 19 19 F F F 1 020-039 9 9 9 16 16 16 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 7 7 7 12 12 12 F F F 4 080-099 5 5 5 5 5 5 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 11 11 11 19 19 19 F F F 7 140-159 6 6 6 11 11 11 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 7 7 12 12 12 F F F 13 260-279 11 11 11 19 19 19 F F F 14 280-299 11 11 11 19 19 19 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 5 5 5 5 5 5 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 10 10 10 17 17 17 F F F 19 380-399 8 8 8 13 13 13 F F F 20 400-419 11 11 11 19 19 19 F F F 21 420-439 10 10 10 17 17 17 F F F 22 440-459 8 8 8 14 14 14 F F F 23 460-479 8 8 8 14 14 14 F F F 24 480-499 10 10 10 18 18 18 F F F 25 500-519 6 6 6 7 7 7 F F F 26 520-539 10 10 10 18 18 18 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 10 10 10 18 18 18 F F F Max gen: 11, Max flush: 19 Begin loop 6 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.032377696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:38.053368779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.05347019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.072148459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.072626813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.072699095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.072732792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.072770236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"Crucible 37feba60-2f37-404d-870d-68fa63e21bd7 has session id: 95339ce9-b8e4-49ea-b571-592b8ea1a5f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.073970695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Upstairs opts: Upstairs UUID: 37feba60-2f37-404d-870d-68fa63e21bd7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074036454Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Crucible stats registered with UUID: 37feba60-2f37-404d-870d-68fa63e21bd7","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074067295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:38.074093627Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074496622Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074553996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074586069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.074919246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.07495681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"37feba60-2f37-404d-870d-68fa63e21bd7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075106225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075138559Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"0","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.07516936Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075208096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075239088Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.07527074Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075395155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:59705)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075437529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37822)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075470815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:35544)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.075584579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.077436644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.077489989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.078318885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.078429062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"0","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.078465314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[0]R flush_numbers[0..12]: [19, 16, 16, 12, 5, 16, 19, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079246075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[0]R generation[0..12]: [11, 9, 9, 7, 5, 9, 11, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079286474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079362364Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[1]R flush_numbers[0..12]: [19, 16, 16, 12, 5, 16, 19, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079439236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[1]R generation[0..12]: [11, 9, 9, 7, 5, 9, 11, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079488663Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079538943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[2]R flush_numbers[0..12]: [19, 16, 16, 12, 5, 16, 19, 11, 1, 12, 16, 10]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T05:59:38.079585014Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[2]R generation[0..12]: [11, 9, 9, 7, 5, 9, 11, 6, 1, 7, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079636645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.07968939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079732174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079773856Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Next flush: 20","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079815228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.07985667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"":"downstairs","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079898763Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079938481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"0","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.079988871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"1","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080032266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"client":"2","":"downstairs","session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080075642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080124618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"37feba60-2f37-404d-870d-68fa63e21bd7 is now active with session: 95339ce9-b8e4-49ea-b571-592b8ea1a5f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080173234Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.08021149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240,"session_id":"95339ce9-b8e4-49ea-b571-592b8ea1a5f5"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080251679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} {"msg":"Activated sub_volume 37feba60-2f37-404d-870d-68fa63e21bd7","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:38.080288151Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 253 len 28672 data: 2 3 3 3 3 4 4 02/30 Read block 351 len 24576 03/30 Write block 333 len 16384 data: 2 2 2 2 04/30 Read block 166 len 24576 05/30 Write block 329 len 16384 data: 3 3 3 3 06/30 Write block 273 len 8192 data: 3 2 07/30 Read block 203 len 16384 08/30 Read block 512 len 24576 09/30 Write block 141 len 12288 data: 2 2 2 10/30 Write block 253 len 40960 data: 3 4 4 4 4 5 5 3 2 2 11/30 Flush 12/30 Write block 275 len 32768 data: 3 3 3 3 3 3 3 2 13/30 Read block 342 len 24576 14/30 Read block 299 len 24576 15/30 Write block 490 len 4096 data: 3 16/30 Read block 286 len 24576 17/30 Read block 294 len 8192 18/30 Read block 198 len 12288 19/30 Write block 468 len 16384 data: 2 2 2 2 20/30 Flush 21/30 Write block 27 len 36864 data: 2 2 3 3 3 3 2 3 3 22/30 Write block 81 len 24576 data: 4 4 4 3 4 4 23/30 Write block 588 len 24576 data: 2 2 2 2 2 2 24/30 Write block 196 len 12288 data: 3 3 2 25/30 Write block 277 len 36864 data: 4 4 4 4 4 3 3 3 3 26/30 Write block 76 len 40960 data: 3 3 3 3 3 5 5 5 4 5 27/30 Read block 92 len 16384 28/30 Write block 256 len 16384 data: 5 5 6 6 29/30 Write block 483 len 20480 data: 3 3 2 2 2 30/30 Read block 503 len 4096 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 4 Sent Done Sent false 1028 Acked Write 5 Sent Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 3 29 Done 4 30 27 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:38.108363895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1240} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 11 11 11 19 19 19 F F F 1 020-039 9 9 9 16 16 16 F T T <--- 2 040-059 9 9 9 16 16 16 F F F 3 060-079 7 7 7 12 12 12 F T T <--- 4 080-099 5 5 5 5 5 5 F T T <--- 5 100-119 9 9 9 16 16 16 F F F 6 120-139 11 11 11 19 19 19 F F F 7 140-159 6 12 12 11 20 20 F F F <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 12 12 12 F T T <--- 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 7 12 12 12 20 20 T T T <--- 13 260-279 11 12 12 19 21 21 F T T <--- 14 280-299 11 12 12 19 21 21 F T T <--- 15 300-319 11 11 11 19 19 19 F F F 16 320-339 5 12 12 5 20 20 T F F <--- 17 340-359 9 9 9 16 16 16 F F F 18 360-379 10 10 10 17 17 17 F F F 19 380-399 8 8 8 13 13 13 F F F 20 400-419 11 11 11 19 19 19 F F F 21 420-439 10 10 10 17 17 17 F F F 22 440-459 8 8 8 14 14 14 F F F 23 460-479 8 12 12 14 21 21 F F F <--- 24 480-499 10 12 12 18 21 21 F T T <--- 25 500-519 6 6 6 7 7 7 F F F 26 520-539 10 10 10 18 18 18 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 10 10 10 18 18 18 F T T <--- Max gen: 12, Max flush: 21 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 11 11 11 19 19 19 F F F 1 020-039 13 13 13 22 22 22 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 13 13 13 22 22 22 F F F 4 080-099 13 13 13 22 22 22 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 11 11 11 19 19 19 F F F 7 140-159 12 12 12 20 20 20 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 13 13 13 22 22 22 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 13 13 13 22 22 22 F F F 13 260-279 13 13 13 22 22 22 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 12 12 12 20 20 20 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 10 10 10 17 17 17 F F F 19 380-399 8 8 8 13 13 13 F F F 20 400-419 11 11 11 19 19 19 F F F 21 420-439 10 10 10 17 17 17 F F F 22 440-459 8 8 8 14 14 14 F F F 23 460-479 12 12 12 21 21 21 F F F 24 480-499 13 13 13 22 22 22 F F F 25 500-519 6 6 6 7 7 7 F F F 26 520-539 10 10 10 18 18 18 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 13, Max flush: 22 Begin loop 7 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.697335309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.726288043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.726860763Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.726956091Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.726987794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.727032653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"Crucible 2a9d24f0-9ebb-439c-8284-ecf7c2fd9937 has session id: 4e4636fc-04bd-4d9b-a05d-63cc6789a8b4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.727980182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Upstairs opts: Upstairs UUID: 2a9d24f0-9ebb-439c-8284-ecf7c2fd9937, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.728048157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Crucible stats registered with UUID: 2a9d24f0-9ebb-439c-8284-ecf7c2fd9937","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.728081252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:42.728109968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.728586579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.728647629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.728679903Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729187445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729223756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"2a9d24f0-9ebb-439c-8284-ecf7c2fd9937 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729542826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729645278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729679566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729713052Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729743181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729891895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.729932224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:57124)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.730033334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:50606)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.730071449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:49348)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.730105066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.73163242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.731828817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.732354865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.732408691Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.732690768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[0]R flush_numbers[0..12]: [19, 22, 16, 22, 22, 16, 19, 20, 1, 22, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.73325417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[0]R generation[0..12]: [11, 13, 9, 13, 13, 9, 11, 12, 1, 13, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733290281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733323557Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[1]R flush_numbers[0..12]: [19, 22, 16, 22, 22, 16, 19, 20, 1, 22, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733354468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[1]R generation[0..12]: [11, 13, 9, 13, 13, 9, 11, 12, 1, 13, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733385148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733417813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[2]R flush_numbers[0..12]: [19, 22, 16, 22, 22, 16, 19, 20, 1, 22, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733448243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[2]R generation[0..12]: [11, 13, 9, 13, 13, 9, 11, 12, 1, 13, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733478252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733510947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733537439Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733571877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Next flush: 23","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733598399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733624721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"":"downstairs","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733660221Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733686253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"0","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733714037Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"1","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733741862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"client":"2","":"downstairs","session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733770338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.7337967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"2a9d24f0-9ebb-439c-8284-ecf7c2fd9937 is now active with session: 4e4636fc-04bd-4d9b-a05d-63cc6789a8b4","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733826419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733851358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248,"session_id":"4e4636fc-04bd-4d9b-a05d-63cc6789a8b4"} {"msg":"Wait for a query_work_queue command to finish before sending IO The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733888351Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} {"msg":"Activated sub_volume 2a9d24f0-9ebb-439c-8284-ecf7c2fd9937","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:42.733912449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 377 len 16384 data: 2 3 4 3 02/30 Write block 379 len 28672 data: 5 4 3 3 3 3 3 03/30 Read block 465 len 28672 04/30 Read block 418 len 12288 05/30 Flush 06/30 Write block 434 len 32768 data: 4 4 4 3 2 2 2 2 07/30 Read block 263 len 36864 08/30 Read block 245 len 16384 09/30 Flush 10/30 Write block 448 len 4096 data: 4 11/30 Read block 153 len 32768 12/30 Read block 580 len 28672 13/30 Flush 14/30 Write block 466 len 40960 data: 2 2 3 3 3 3 2 2 2 2 15/30 Read block 575 len 32768 16/30 Read block 46 len 12288 17/30 Write block 145 len 16384 data: 2 3 3 3 18/30 Read block 61 len 12288 19/30 Read block 99 len 20480 20/30 Write block 11 len 36864 data: 4 4 4 3 3 2 3 3 3 21/30 Write block 334 len 20480 data: 3 3 3 2 2 22/30 Write block 166 len 40960 data: 2 2 2 2 2 2 2 2 2 2 23/30 Write block 73 len 24576 data: 2 2 2 4 4 4 24/30 Write block 499 len 24576 data: 2 2 2 2 2 3 25/30 Read block 176 len 24576 26/30 Write block 465 len 4096 data: 2 27/30 Write block 417 len 32768 data: 2 2 3 4 4 4 4 4 28/30 Write block 483 len 28672 data: 4 4 3 3 3 2 3 29/30 Write block 183 len 24576 data: 2 2 2 2 2 2 30/30 Read block 535 len 36864 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 3 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Write 6 Done Sent Done false 1029 Acked Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:42.763652236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1248} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 11 11 11 19 19 19 T F T <--- 1 020-039 13 13 13 22 22 22 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 13 13 13 22 22 22 T F T <--- 4 080-099 13 13 13 22 22 22 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 11 11 11 19 19 19 F F F 7 140-159 12 12 12 20 20 20 T F T <--- 8 160-179 1 1 1 1 1 1 T F T <--- 9 180-199 13 13 13 22 22 22 T F T <--- 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 13 13 13 22 22 22 F F F 13 260-279 13 13 13 22 22 22 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 12 12 12 20 20 20 T F T <--- 17 340-359 9 9 9 16 16 16 F F F 18 360-379 14 10 14 23 17 23 F T F <--- 19 380-399 14 8 14 23 13 23 F T F <--- 20 400-419 11 11 11 19 19 19 T F T <--- 21 420-439 14 10 14 24 17 24 T F T <--- 22 440-459 14 8 14 25 14 25 F F F <--- 23 460-479 12 12 12 21 21 21 T F T <--- 24 480-499 13 13 13 22 22 22 T F T <--- 25 500-519 6 6 6 7 7 7 T F T <--- 26 520-539 10 10 10 18 18 18 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 14, Max flush: 25 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 13 13 13 22 22 22 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 15 15 15 26 26 26 F F F 4 080-099 13 13 13 22 22 22 F F F 5 100-119 9 9 9 16 16 16 F F F 6 120-139 11 11 11 19 19 19 F F F 7 140-159 15 15 15 26 26 26 F F F 8 160-179 15 15 15 26 26 26 F F F 9 180-199 15 15 15 26 26 26 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 13 13 13 22 22 22 F F F 13 260-279 13 13 13 22 22 22 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 15 15 15 26 26 26 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 15 15 15 26 26 26 F F F 21 420-439 15 15 15 26 26 26 F F F 22 440-459 14 14 14 25 25 25 F F F 23 460-479 15 15 15 26 26 26 F F F 24 480-499 15 15 15 26 26 26 F F F 25 500-519 15 15 15 26 26 26 F F F 26 520-539 10 10 10 18 18 18 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 15, Max flush: 26 Begin loop 8 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.356809672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.379426333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.379993542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.380146854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.380204548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.380266541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"Crucible 432f6a96-035d-4a60-8b27-97b813d30fe3 has session id: 2c85db97-0796-4b94-9a35-680e1f23a886","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.381285401Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Upstairs opts: Upstairs UUID: 432f6a96-035d-4a60-8b27-97b813d30fe3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.381354808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Crucible stats registered with UUID: 432f6a96-035d-4a60-8b27-97b813d30fe3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.38138624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:47.381412993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.381870826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.381955984Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.381982617Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382251498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382290214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"432f6a96-035d-4a60-8b27-97b813d30fe3 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382579775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382663521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.38269334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382785551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382834187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382861772Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.382888074Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"ds_connection connected from Ok(127.0.0.1:39668)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.383218636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"ds_connection connected from Ok(127.0.0.1:32917)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.38327623Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"ds_connection connected from Ok(127.0.0.1:54834)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.383319806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.385639193Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.385724361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.385937582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.386420705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.386516414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[0]R flush_numbers[0..12]: [26, 22, 16, 26, 22, 16, 19, 26, 26, 26, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387322835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[0]R generation[0..12]: [15, 13, 9, 15, 13, 9, 11, 15, 15, 15, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387362954Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387390358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[1]R flush_numbers[0..12]: [26, 22, 16, 26, 22, 16, 19, 26, 26, 26, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387415027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[1]R generation[0..12]: [15, 13, 9, 15, 13, 9, 11, 15, 15, 15, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387445628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387471609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[2]R flush_numbers[0..12]: [26, 22, 16, 26, 22, 16, 19, 26, 26, 26, 16, 10]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387496348Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[2]R generation[0..12]: [15, 13, 9, 15, 13, 9, 11, 15, 15, 15, 9, 6]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387520115Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387546036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387566907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.38758872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Next flush: 27","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387609621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387630613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"":"downstairs","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387652516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387672565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"0","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387697725Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"1","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387719528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"client":"2","":"downstairs","session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387741902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T05:59:47.387763004Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"432f6a96-035d-4a60-8b27-97b813d30fe3 is now active with session: 2c85db97-0796-4b94-9a35-680e1f23a886","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387798233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387818393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256,"session_id":"2c85db97-0796-4b94-9a35-680e1f23a886"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.387839264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} {"msg":"Activated sub_volume 432f6a96-035d-4a60-8b27-97b813d30fe3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:47.38785753Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 156 len 4096 data: 3 02/30 Write block 412 len 28672 data: 2 2 3 3 3 3 3 03/30 Write block 125 len 16384 data: 3 3 3 3 04/30 Read block 20 len 12288 05/30 Write block 113 len 24576 data: 3 2 4 4 3 3 06/30 Read block 207 len 8192 07/30 Read block 386 len 4096 08/30 Write block 249 len 20480 data: 2 2 2 2 4 09/30 Write block 256 len 36864 data: 6 6 7 7 4 3 3 2 2 10/30 Write block 242 len 12288 data: 3 3 3 11/30 Write block 522 len 4096 data: 4 12/30 Write block 280 len 40960 data: 5 5 4 4 4 4 3 3 3 3 13/30 Read block 244 len 12288 14/30 Write block 500 len 20480 data: 3 3 3 3 4 15/30 Write block 139 len 12288 data: 3 3 3 16/30 Write block 126 len 36864 data: 4 4 4 3 2 2 2 2 2 17/30 Write block 282 len 32768 data: 5 5 5 5 4 4 4 4 18/30 Write block 142 len 16384 data: 3 3 2 3 19/30 Read block 175 len 8192 20/30 Write block 237 len 8192 data: 2 2 21/30 Write block 183 len 20480 data: 3 3 3 3 3 22/30 Write block 20 len 8192 data: 3 3 23/30 Write block 75 len 4096 data: 3 24/30 Write block 76 len 32768 data: 5 5 5 4 4 6 6 6 25/30 Write block 277 len 24576 data: 5 5 5 6 6 6 26/30 Flush 27/30 Write block 332 len 20480 data: 4 3 4 4 4 28/30 Write block 437 len 4096 data: 4 29/30 Read block 433 len 28672 30/30 Read block 456 len 16384 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:47.409724261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1256} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 13 27 27 22 F F F <--- 2 040-059 9 9 9 16 16 16 F F F 3 060-079 16 16 15 27 27 26 F F F <--- 4 080-099 16 16 13 27 27 22 F F F <--- 5 100-119 16 16 9 27 27 16 F F F <--- 6 120-139 16 16 11 27 27 19 F F F <--- 7 140-159 16 16 15 27 27 26 F F T <--- 8 160-179 15 15 15 26 26 26 F F F 9 180-199 16 16 15 27 27 26 F F F <--- 10 200-219 9 9 9 16 16 16 F F F 11 220-239 16 16 6 27 27 10 F F F <--- 12 240-259 16 16 13 27 27 22 F F F <--- 13 260-279 16 16 13 27 27 22 F F F <--- 14 280-299 16 16 13 27 27 22 F F F <--- 15 300-319 11 11 11 19 19 19 F F F 16 320-339 15 15 15 26 26 26 T T F <--- 17 340-359 9 9 9 16 16 16 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 16 16 15 27 27 26 F F T <--- 21 420-439 15 15 15 26 26 26 T T F <--- 22 440-459 14 14 14 25 25 25 F F F 23 460-479 15 15 15 26 26 26 F F F 24 480-499 15 15 15 26 26 26 F F F 25 500-519 16 16 15 27 27 26 F F F <--- 26 520-539 16 16 10 27 27 18 F F F <--- 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 16, Max flush: 27 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 16 16 16 27 27 27 F F F 4 080-099 16 16 16 27 27 27 F F F 5 100-119 16 16 16 27 27 27 F F F 6 120-139 16 16 16 27 27 27 F F F 7 140-159 16 16 16 27 27 27 F F F 8 160-179 15 15 15 26 26 26 F F F 9 180-199 16 16 16 27 27 27 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 16 16 16 27 27 27 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 17 17 17 28 28 28 F F F 17 340-359 9 9 9 16 16 16 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 16 16 16 27 27 27 F F F 21 420-439 17 17 17 28 28 28 F F F 22 440-459 14 14 14 25 25 25 F F F 23 460-479 15 15 15 26 26 26 F F F 24 480-499 15 15 15 26 26 26 F F F 25 500-519 16 16 16 27 27 27 F F F 26 520-539 16 16 16 27 27 27 F F F 27 540-559 10 10 10 18 18 18 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 17, Max flush: 28 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.014140974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.035272965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.035773653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.035894311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.035932737Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.035985521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"Crucible 7580fc1d-adbb-494b-8280-b9f76ef7535e has session id: 5196b67a-a854-451d-b6e4-b930b9a4f4bd","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.036913132Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Upstairs opts: Upstairs UUID: 7580fc1d-adbb-494b-8280-b9f76ef7535e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.037004572Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Crucible stats registered with UUID: 7580fc1d-adbb-494b-8280-b9f76ef7535e","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.037059621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:52.037103938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.037490751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.037557764Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.037590889Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038209309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038327202Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"7580fc1d-adbb-494b-8280-b9f76ef7535e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038372642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038416148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038462389Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038502698Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038542828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038586364Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038618918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:54377)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.038710258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:48463)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.039186037Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:62665)","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.039267648Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.040877996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.041353564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.041814454Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.041900273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042249091Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[0]R flush_numbers[0..12]: [26, 27, 16, 27, 27, 27, 27, 27, 26, 27, 16, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.04275492Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[0]R generation[0..12]: [15, 16, 9, 16, 16, 16, 16, 16, 15, 16, 9, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042791081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042825549Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[1]R flush_numbers[0..12]: [26, 27, 16, 27, 27, 27, 27, 27, 26, 27, 16, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042857863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[1]R generation[0..12]: [15, 16, 9, 16, 16, 16, 16, 16, 15, 16, 9, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042889185Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.04292257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[2]R flush_numbers[0..12]: [26, 27, 16, 27, 27, 27, 27, 27, 26, 27, 16, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042953852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[2]R generation[0..12]: [15, 16, 9, 16, 16, 16, 16, 16, 15, 16, 9, 16]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.042984292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043017528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043045182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043073258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Next flush: 29","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043107034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043133998Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"":"downstairs","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043162213Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043187974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"0","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2026-03-06T05:59:52.043228444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"1","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043258233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"client":"2","":"downstairs","session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.04328711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043313833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"7580fc1d-adbb-494b-8280-b9f76ef7535e is now active with session: 5196b67a-a854-451d-b6e4-b930b9a4f4bd","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043344062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043369623Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264,"session_id":"5196b67a-a854-451d-b6e4-b930b9a4f4bd"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043396225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} {"msg":"Activated sub_volume 7580fc1d-adbb-494b-8280-b9f76ef7535e","v":0,"name":"crucible","level":30,"time":"2026-03-06T05:59:52.043419721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 230 len 36864 data: 3 3 3 2 2 2 2 3 3 02/30 Write block 536 len 28672 data: 4 4 3 2 2 2 2 03/30 Write block 63 len 24576 data: 2 2 2 2 2 3 04/30 Write block 517 len 28672 data: 2 2 3 4 4 5 4 05/30 Read block 238 len 8192 06/30 Read block 67 len 4096 07/30 Read block 330 len 24576 08/30 Read block 47 len 16384 09/30 Read block 75 len 28672 10/30 Read block 120 len 16384 11/30 Write block 157 len 32768 data: 3 2 2 2 2 2 2 2 12/30 Read block 524 len 8192 13/30 Write block 423 len 36864 data: 5 5 3 2 2 3 3 4 4 14/30 Read block 82 len 4096 15/30 Read block 422 len 28672 16/30 Read block 7 len 28672 17/30 Write block 112 len 8192 data: 3 4 18/30 Read block 478 len 28672 19/30 Write block 72 len 40960 data: 2 3 3 4 6 6 6 5 5 7 20/30 Write block 446 len 8192 data: 4 4 21/30 Read block 31 len 4096 22/30 Read block 184 len 28672 23/30 Write block 570 len 32768 data: 4 2 2 2 2 2 2 2 24/30 Read block 50 len 40960 25/30 Read block 275 len 32768 26/30 Write block 490 len 36864 data: 4 3 3 3 3 2 2 2 2 27/30 Read block 430 len 16384 28/30 Write block 174 len 40960 data: 3 3 2 2 2 2 2 2 2 4 29/30 Write block 345 len 16384 data: 2 2 2 2 30/30 Read block 501 len 16384 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 25 0 26 Done 29 5 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T05:59:52.079262693Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1264} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 16 16 16 27 27 27 T T T 4 080-099 16 16 16 27 27 27 T F T <--- 5 100-119 16 16 16 27 27 27 T F T <--- 6 120-139 16 16 16 27 27 27 F F F 7 140-159 16 16 16 27 27 27 T F T <--- 8 160-179 15 15 15 26 26 26 T F T <--- 9 180-199 16 16 16 27 27 27 T F T <--- 10 200-219 9 9 9 16 16 16 F F F 11 220-239 16 16 16 27 27 27 T T T 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 17 17 17 28 28 28 F F F 17 340-359 9 9 9 16 16 16 T F T <--- 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 16 16 16 27 27 27 F F F 21 420-439 17 17 17 28 28 28 T F T <--- 22 440-459 14 14 14 25 25 25 T F T <--- 23 460-479 15 15 15 26 26 26 F F F 24 480-499 15 15 15 26 26 26 T F T <--- 25 500-519 16 16 16 27 27 27 T T T 26 520-539 16 16 16 27 27 27 T T T 27 540-559 10 10 10 18 18 18 T T T 28 560-579 11 11 11 19 19 19 T F T <--- 29 580-599 13 13 13 22 22 22 F F F Max gen: 17, Max flush: 28 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 9 9 9 16 16 16 F F F 3 060-079 18 18 18 29 29 29 F F F 4 080-099 18 18 18 29 29 29 F F F 5 100-119 18 18 18 29 29 29 F F F 6 120-139 16 16 16 27 27 27 F F F 7 140-159 18 18 18 29 29 29 F F F 8 160-179 18 18 18 29 29 29 F F F 9 180-199 18 18 18 29 29 29 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 11 11 11 19 19 19 F F F 16 320-339 17 17 17 28 28 28 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 16 16 16 27 27 27 F F F 21 420-439 18 18 18 29 29 29 F F F 22 440-459 18 18 18 29 29 29 F F F 23 460-479 15 15 15 26 26 26 F F F 24 480-499 18 18 18 29 29 29 F F F 25 500-519 18 18 18 29 29 29 F F F 26 520-539 18 18 18 29 29 29 F F F 27 540-559 18 18 18 29 29 29 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 18, Max flush: 29 Begin loop 10 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.668670173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.696766357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.69739612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.697509654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.697545114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.697583931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"Crucible b5cbcfa2-51a0-4249-84f9-a1604c031669 has session id: 4cd82154-c324-4b1c-af76-3bf5c435c4e0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.69840971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Upstairs opts: Upstairs UUID: b5cbcfa2-51a0-4249-84f9-a1604c031669, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.698539607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Crucible stats registered with UUID: b5cbcfa2-51a0-4249-84f9-a1604c031669","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.6985718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:06.698593112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.698935468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.698987621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699011879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699463249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699538408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"b5cbcfa2-51a0-4249-84f9-a1604c031669 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699611903Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699636281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699658104Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699679336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699700017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699738543Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42456)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.6998892Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.699918137Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:36918)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70049234Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:55793)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70053295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.702019394Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.702805745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.702902556Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70292959Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.703818634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[0]R flush_numbers[0..12]: [26, 27, 16, 29, 29, 29, 27, 29, 29, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704095591Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[0]R generation[0..12]: [15, 16, 9, 18, 18, 18, 16, 18, 18, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704146631Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704179246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[1]R flush_numbers[0..12]: [26, 27, 16, 29, 29, 29, 27, 29, 29, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704208614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[1]R generation[0..12]: [15, 16, 9, 18, 18, 18, 16, 18, 18, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704249274Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704287028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[2]R flush_numbers[0..12]: [26, 27, 16, 29, 29, 29, 27, 29, 29, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704320525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[2]R generation[0..12]: [15, 16, 9, 18, 18, 18, 16, 18, 18, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704349341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704385413Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Max found gen is 19","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704411183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Generation requested: 20 >= found:19","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70443986Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704466382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704496341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"":"downstairs","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704523385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704551941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"0","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704579105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"1","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:00:06.704604735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"client":"2","":"downstairs","session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70465262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704686477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"b5cbcfa2-51a0-4249-84f9-a1604c031669 is now active with session: 4cd82154-c324-4b1c-af76-3bf5c435c4e0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704724923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704758759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272,"session_id":"4cd82154-c324-4b1c-af76-3bf5c435c4e0"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.70478464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} {"msg":"Activated sub_volume b5cbcfa2-51a0-4249-84f9-a1604c031669","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:06.704808487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 85 len 12288 data: 6 5 4 02/30 Read block 292 len 36864 03/30 Read block 160 len 32768 04/30 Write block 319 len 12288 data: 2 2 2 05/30 Flush 06/30 Write block 169 len 16384 data: 3 3 3 3 07/30 Read block 28 len 12288 08/30 Write block 130 len 16384 data: 3 3 3 3 09/30 Write block 496 len 4096 data: 3 10/30 Flush 11/30 Write block 54 len 32768 data: 2 2 2 2 2 2 2 2 12/30 Read block 117 len 36864 13/30 Read block 500 len 40960 14/30 Read block 138 len 32768 15/30 Write block 473 len 32768 data: 3 3 3 2 2 2 2 2 16/30 Flush 17/30 Write block 497 len 28672 data: 3 3 3 4 4 4 4 18/30 Write block 403 len 20480 data: 5 4 4 4 3 19/30 Read block 77 len 20480 20/30 Write block 544 len 24576 data: 2 2 2 3 3 3 21/30 Read block 548 len 12288 22/30 Read block 469 len 8192 23/30 Read block 410 len 28672 24/30 Write block 509 len 20480 data: 3 3 3 2 2 25/30 Read block 292 len 24576 26/30 Read block 91 len 4096 27/30 Write block 61 len 36864 data: 3 2 3 3 3 3 3 4 2 28/30 Read block 117 len 28672 29/30 Write block 75 len 32768 data: 5 7 7 7 6 6 8 7 30/30 Write block 88 len 8192 data: 4 4 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Read 8 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Read 3 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:06.737980498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1272} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 20 20 9 32 32 16 F F F <--- 3 060-079 20 20 18 32 32 29 T T F <--- 4 080-099 20 20 18 30 30 29 T T T <--- 5 100-119 18 18 18 29 29 29 F F F 6 120-139 20 20 16 31 31 27 F F F <--- 7 140-159 18 18 18 29 29 29 F F F 8 160-179 20 20 18 31 31 29 F F F <--- 9 180-199 18 18 18 29 29 29 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 20 20 11 30 30 19 F F F <--- 16 320-339 20 20 17 30 30 28 F F F <--- 17 340-359 18 18 18 29 29 29 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 16 16 16 27 27 27 T T F <--- 21 420-439 18 18 18 29 29 29 F F F 22 440-459 18 18 18 29 29 29 F F F 23 460-479 20 20 15 32 32 26 F F F <--- 24 480-499 20 20 18 32 32 29 T T F <--- 25 500-519 18 18 18 29 29 29 T T F <--- 26 520-539 18 18 18 29 29 29 F F F 27 540-559 18 18 18 29 29 29 T T F <--- 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 20, Max flush: 32 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 20 20 20 32 32 32 F F F 3 060-079 21 21 21 33 33 33 F F F 4 080-099 21 21 21 33 33 33 F F F 5 100-119 18 18 18 29 29 29 F F F 6 120-139 20 20 20 31 31 31 F F F 7 140-159 18 18 18 29 29 29 F F F 8 160-179 20 20 20 31 31 31 F F F 9 180-199 18 18 18 29 29 29 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 20 20 20 30 30 30 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 21 21 21 33 33 33 F F F 21 420-439 18 18 18 29 29 29 F F F 22 440-459 18 18 18 29 29 29 F F F 23 460-479 20 20 20 32 32 32 F F F 24 480-499 21 21 21 33 33 33 F F F 25 500-519 21 21 21 33 33 33 F F F 26 520-539 18 18 18 29 29 29 F F F 27 540-559 21 21 21 33 33 33 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 21, Max flush: 33 Begin loop 11 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.378568707Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.403361131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.403977608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.404070681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.404099879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.404144878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"Crucible 45f8e227-8b08-442f-b698-a9052c5a9e2d has session id: 739b9b2f-9bb3-4759-946d-19d705becadb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405134801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Upstairs opts: Upstairs UUID: 45f8e227-8b08-442f-b698-a9052c5a9e2d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405210611Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Crucible stats registered with UUID: 45f8e227-8b08-442f-b698-a9052c5a9e2d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405247353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:21.405279076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405719155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405774133Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.405816487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406207017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406286464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"45f8e227-8b08-442f-b698-a9052c5a9e2d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406317565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406350971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406392313Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.4064316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406467131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406502941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406804867Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"ds_connection connected from Ok(127.0.0.1:60371)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406915445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"ds_connection connected from Ok(127.0.0.1:41154)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.406960524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:21.407365614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:21.407436513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:21.40748549Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.407834208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.407896261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.409089525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.409362394Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.409771922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:21.410168695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.409694132Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"ds_connection connected from Ok(127.0.0.1:52496)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.40981987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.412625887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[0]R flush_numbers[0..12]: [26, 27, 32, 33, 33, 29, 31, 29, 31, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414251705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[0]R generation[0..12]: [15, 16, 20, 21, 21, 18, 20, 18, 20, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414327234Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414381211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[1]R flush_numbers[0..12]: [26, 27, 32, 33, 33, 29, 31, 29, 31, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414427292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[1]R generation[0..12]: [15, 16, 20, 21, 21, 18, 20, 18, 20, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.41446683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414511558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[2]R flush_numbers[0..12]: [26, 27, 32, 33, 33, 29, 31, 29, 31, 29, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414565034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[2]R generation[0..12]: [15, 16, 20, 21, 21, 18, 20, 18, 20, 18, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414606756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414663718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414741171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414784166Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Next flush: 34","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414832231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.41487755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"":"downstairs","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414919503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.414957648Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"0","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-06T06:00:31.415033779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"1","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415096442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"client":"2","":"downstairs","session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415141982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415185769Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"45f8e227-8b08-442f-b698-a9052c5a9e2d is now active with session: 739b9b2f-9bb3-4759-946d-19d705becadb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415240236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415288211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280,"session_id":"739b9b2f-9bb3-4759-946d-19d705becadb"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415337338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} {"msg":"Activated sub_volume 45f8e227-8b08-442f-b698-a9052c5a9e2d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:31.415387316Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 bmat-EVT22200007-0000ec3a","pid":1280} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 554 len 12288 data: 2 2 2 02/30 Read block 512 len 20480 03/30 Read block 332 len 28672 04/30 Read block 333 len 32768 05/30 Write block 310 len 16384 data: 2 2 2 2 06/30 Read block 255 len 36864 07/30 Read block 532 len 24576 08/30 Read block 267 len 32768 09/30 Read block 405 len 8192 10/30 Flush 11/30 Write block 63 len 20480 data: 4 4 4 4 4 12/30 Write block 180 len 16384 data: 3 3 3 5 13/30 Read block 319 len 20480 14/30 Read block 224 len 36864 15/30 Read block 519 len 20480 16/30 Read block 113 len 40960 17/30 Write block 514 len 20480 data: 2 2 2 3 3 18/30 Read block 464 len 40960 19/30 Read block 112 len 28672 20/30 Read block 109 len 40960 21/30 Flush 22/30 Flush 23/30 Write block 272 len 8192 data: 4 4 24/30 Read block 315 len 12288 25/30 Read block 327 len 32768 26/30 Read block 13 len 32768 27/30 Read block 221 len 28672 28/30 Read block 364 len 24576 29/30 Write block 524 len 4096 data: 4 30/30 Read block 83 len 40960 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 5 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:31.450384538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1280} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 20 20 20 32 32 32 F F F 3 060-079 22 21 22 35 33 35 F F F <--- 4 080-099 21 21 21 33 33 33 F F F 5 100-119 18 18 18 29 29 29 F F F 6 120-139 20 20 20 31 31 31 F F F 7 140-159 18 18 18 29 29 29 F F F 8 160-179 20 20 20 31 31 31 F F F 9 180-199 22 18 22 35 29 35 F F F <--- 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 16 16 16 27 27 27 T F T <--- 14 280-299 16 16 16 27 27 27 F F F 15 300-319 22 20 22 34 30 34 F F F <--- 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 21 21 21 33 33 33 F F F 21 420-439 18 18 18 29 29 29 F F F 22 440-459 18 18 18 29 29 29 F F F 23 460-479 20 20 20 32 32 32 F F F 24 480-499 21 21 21 33 33 33 F F F 25 500-519 22 21 22 35 33 35 F F F <--- 26 520-539 18 18 18 29 29 29 T F T <--- 27 540-559 22 21 22 34 33 34 F T F <--- 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 22, Max flush: 35 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 20 20 20 32 32 32 F F F 3 060-079 22 22 22 35 35 35 F F F 4 080-099 21 21 21 33 33 33 F F F 5 100-119 18 18 18 29 29 29 F F F 6 120-139 20 20 20 31 31 31 F F F 7 140-159 18 18 18 29 29 29 F F F 8 160-179 20 20 20 31 31 31 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 23 23 23 36 36 36 F F F 14 280-299 16 16 16 27 27 27 F F F 15 300-319 22 22 22 34 34 34 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 14 14 14 23 23 23 F F F 19 380-399 14 14 14 23 23 23 F F F 20 400-419 21 21 21 33 33 33 F F F 21 420-439 18 18 18 29 29 29 F F F 22 440-459 18 18 18 29 29 29 F F F 23 460-479 20 20 20 32 32 32 F F F 24 480-499 21 21 21 33 33 33 F F F 25 500-519 22 22 22 35 35 35 F F F 26 520-539 23 23 23 36 36 36 F F F 27 540-559 22 22 22 34 34 34 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 23, Max flush: 36 Begin loop 12 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:35.983502974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.007025963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.007754912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.007830341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.007859057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.007898054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"Crucible 34e0979b-6953-418b-beb6-a70e8a3d78b3 has session id: 7effccfd-7bb3-4294-943e-ff4a30cb9111","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009174784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Upstairs opts: Upstairs UUID: 34e0979b-6953-418b-beb6-a70e8a3d78b3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009304069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Crucible stats registered with UUID: 34e0979b-6953-418b-beb6-a70e8a3d78b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009344889Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:36.009378075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009886668Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009958931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.009994481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010435001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010474789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"34e0979b-6953-418b-beb6-a70e8a3d78b3 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010600096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010649183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.01068317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010716135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010751976Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010788789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010825702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"ds_connection connected from Ok(127.0.0.1:44939)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.010901331Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"ds_connection connected from Ok(127.0.0.1:33287)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.011015286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"ds_connection connected from Ok(127.0.0.1:40783)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.011445334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.013215968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.013707719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.014262443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.014310388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.014923488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[0]R flush_numbers[0..12]: [26, 27, 32, 35, 33, 29, 31, 29, 31, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015143793Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[0]R generation[0..12]: [15, 16, 20, 22, 21, 18, 20, 18, 20, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015190194Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015224702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[1]R flush_numbers[0..12]: [26, 27, 32, 35, 33, 29, 31, 29, 31, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015257176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[1]R generation[0..12]: [15, 16, 20, 22, 21, 18, 20, 18, 20, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015288939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015322044Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[2]R flush_numbers[0..12]: [26, 27, 32, 35, 33, 29, 31, 29, 31, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015353076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[2]R generation[0..12]: [15, 16, 20, 22, 21, 18, 20, 18, 20, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015383566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015416561Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015443815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.01547177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015498372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015525135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"":"downstairs","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.01555316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015578911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"0","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015607187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"1","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.01563881Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"client":"2","":"downstairs","session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2026-03-06T06:00:36.015667596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015708467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"34e0979b-6953-418b-beb6-a70e8a3d78b3 is now active with session: 7effccfd-7bb3-4294-943e-ff4a30cb9111","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015738897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015764598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288,"session_id":"7effccfd-7bb3-4294-943e-ff4a30cb9111"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.015807382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} {"msg":"Activated sub_volume 34e0979b-6953-418b-beb6-a70e8a3d78b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:36.0158314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 549 len 8192 data: 4 3 03/30 Write block 162 len 24576 data: 3 3 3 2 3 3 04/30 Write block 458 len 32768 data: 2 2 3 3 3 3 2 3 05/30 Write block 531 len 36864 data: 4 5 6 4 4 5 5 4 3 06/30 Write block 120 len 8192 data: 3 3 07/30 Read block 76 len 20480 08/30 Write block 550 len 40960 data: 4 3 3 2 3 3 3 2 3 4 09/30 Read block 332 len 12288 10/30 Read block 442 len 40960 11/30 Write block 54 len 36864 data: 3 3 3 3 3 3 3 4 3 12/30 Write block 408 len 28672 data: 2 2 2 2 3 3 4 13/30 Write block 378 len 20480 data: 4 6 5 4 4 14/30 Flush 15/30 Write block 9 len 20480 data: 2 3 5 5 5 16/30 Write block 315 len 4096 data: 2 17/30 Write block 376 len 12288 data: 2 3 5 18/30 Write block 127 len 8192 data: 5 5 19/30 Read block 460 len 40960 20/30 Write block 295 len 36864 data: 2 2 3 3 3 3 3 3 3 21/30 Write block 513 len 36864 data: 3 3 3 3 4 4 4 5 5 22/30 Read block 109 len 8192 23/30 Read block 67 len 16384 24/30 Read block 342 len 28672 25/30 Read block 9 len 4096 26/30 Write block 441 len 24576 data: 3 3 3 3 4 5 27/30 Read block 152 len 20480 28/30 Read block 417 len 8192 29/30 Write block 82 len 32768 data: 8 7 5 7 6 5 5 5 30/30 Write block 427 len 4096 data: 3 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 26 30 Done 27 27 3 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:36.043533566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1288} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 26 26 26 T T F <--- 1 020-039 16 16 16 27 27 27 F F F 2 040-059 24 24 20 38 38 32 F F F <--- 3 060-079 24 24 22 38 38 35 F F F <--- 4 080-099 21 21 21 33 33 33 T T F <--- 5 100-119 18 18 18 29 29 29 F F F 6 120-139 24 24 20 38 38 31 T T F <--- 7 140-159 18 18 18 29 29 29 F F F 8 160-179 24 24 20 38 38 31 F F T <--- 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 23 23 23 36 36 36 F F F 14 280-299 16 16 16 27 27 27 T T F <--- 15 300-319 22 22 22 34 34 34 T T F <--- 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 24 24 14 38 38 23 T T F <--- 19 380-399 24 24 14 38 38 23 F F F <--- 20 400-419 24 24 21 38 38 33 F F F <--- 21 420-439 18 18 18 29 29 29 T T F <--- 22 440-459 24 24 18 38 38 29 T T T <--- 23 460-479 24 24 20 38 38 32 F F T <--- 24 480-499 21 21 21 33 33 33 F F F 25 500-519 22 22 22 35 35 35 T T F <--- 26 520-539 24 24 23 38 38 36 T T F <--- 27 540-559 24 24 22 38 38 34 F F T <--- 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 24, Max flush: 38 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 24 24 24 38 38 38 F F F 3 060-079 24 24 24 38 38 38 F F F 4 080-099 25 25 25 39 39 39 F F F 5 100-119 18 18 18 29 29 29 F F F 6 120-139 25 25 25 39 39 39 F F F 7 140-159 18 18 18 29 29 29 F F F 8 160-179 24 24 24 38 38 38 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 23 23 23 36 36 36 F F F 14 280-299 25 25 25 39 39 39 F F F 15 300-319 25 25 25 39 39 39 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 25 25 25 39 39 39 F F F 19 380-399 24 24 24 38 38 38 F F F 20 400-419 24 24 24 38 38 38 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 25 25 25 39 39 39 F F F 23 460-479 24 24 24 38 38 38 F F F 24 480-499 21 21 21 33 33 33 F F F 25 500-519 25 25 25 39 39 39 F F F 26 520-539 25 25 25 39 39 39 F F F 27 540-559 24 24 24 38 38 38 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 25, Max flush: 39 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.67210244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:40.694852344Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.694938814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.713483991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.713935381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.714016451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.714045008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.714084355Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"Crucible 5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9 has session id: 42af3e63-cf44-41fe-8d08-61af575180ee","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.715276247Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Upstairs opts: Upstairs UUID: 5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.715494718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Crucible stats registered with UUID: 5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.71553095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:40.715554216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.7160731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716130192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716156124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716603737Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716709826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716760336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716806948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716852838Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.716950742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.71700571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.717112721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.717155365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:41422)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.717204312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:57704)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.717249672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:50268)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.717465267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.719343674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.720027333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.720154233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.72061894Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.720925445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[0]R flush_numbers[0..12]: [39, 27, 38, 38, 39, 29, 39, 29, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.720972759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[0]R generation[0..12]: [25, 16, 24, 24, 25, 18, 25, 18, 24, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721012507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721053899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[1]R flush_numbers[0..12]: [39, 27, 38, 38, 39, 29, 39, 29, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721092114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[1]R generation[0..12]: [25, 16, 24, 24, 25, 18, 25, 18, 24, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721129638Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721170138Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[2]R flush_numbers[0..12]: [39, 27, 38, 38, 39, 29, 39, 29, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721214606Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[2]R generation[0..12]: [25, 16, 24, 24, 25, 18, 25, 18, 24, 22, 9, 18]","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-03-06T06:00:40.721428358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721474348Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721557943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:00:40.721691136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721772898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721806935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"":"downstairs","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721841343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721872374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"0","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721906521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"1","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721940639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"client":"2","":"downstairs","session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.721975277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.722007661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9 is now active with session: 42af3e63-cf44-41fe-8d08-61af575180ee","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.722044223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.722075284Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296,"session_id":"42af3e63-cf44-41fe-8d08-61af575180ee"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.722107468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} {"msg":"Activated sub_volume 5c917464-c5bc-4b9f-b5e5-6d14b0f8a3d9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:40.722140433Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 312 len 16384 data: 3 3 2 3 03/30 Write block 144 len 32768 data: 3 4 4 4 4 3 3 4 04/30 Read block 74 len 36864 05/30 Read block 151 len 8192 06/30 Read block 104 len 36864 07/30 Read block 65 len 28672 08/30 Read block 532 len 24576 09/30 Write block 45 len 16384 data: 2 2 2 2 10/30 Read block 551 len 20480 11/30 Read block 218 len 16384 12/30 Read block 50 len 12288 13/30 Write block 73 len 40960 data: 4 4 6 8 8 8 7 7 9 9 14/30 Read block 80 len 20480 15/30 Flush 16/30 Write block 487 len 12288 data: 4 3 4 17/30 Read block 7 len 28672 18/30 Flush 19/30 Write block 276 len 28672 data: 4 6 6 6 7 7 7 20/30 Write block 116 len 12288 data: 5 4 4 21/30 Read block 482 len 36864 22/30 Read block 254 len 28672 23/30 Read block 329 len 16384 24/30 Read block 287 len 4096 25/30 Read block 446 len 4096 26/30 Read block 474 len 4096 27/30 Read block 350 len 16384 28/30 Write block 524 len 4096 data: 5 29/30 Write block 83 len 12288 data: 8 6 8 30/30 Read block 32 len 28672 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Sent Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 3 Sent Done Done false 1029 Acked Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 0 29 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:40.753335993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1296} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 24 26 26 38 41 41 F F F <--- 3 060-079 24 26 26 38 41 41 F F F <--- 4 080-099 25 26 26 39 41 41 F T T <--- 5 100-119 18 18 18 29 29 29 F T T <--- 6 120-139 25 25 25 39 39 39 F F F 7 140-159 18 26 26 29 41 41 F F F <--- 8 160-179 24 24 24 38 38 38 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 23 23 23 36 36 36 F T T <--- 14 280-299 25 25 25 39 39 39 F T T <--- 15 300-319 25 26 26 39 41 41 F F F <--- 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 25 25 25 39 39 39 F F F 19 380-399 24 24 24 38 38 38 F F F 20 400-419 24 24 24 38 38 38 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 25 25 25 39 39 39 F F F 23 460-479 24 24 24 38 38 38 F F F 24 480-499 21 26 26 33 42 42 F F F <--- 25 500-519 25 25 25 39 39 39 F F F 26 520-539 25 25 25 39 39 39 F T T <--- 27 540-559 24 24 24 38 38 38 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 26, Max flush: 42 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 25 25 25 39 39 39 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 24 24 24 38 38 38 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 F F F 12 240-259 16 16 16 27 27 27 F F F 13 260-279 27 27 27 43 43 43 F F F 14 280-299 27 27 27 43 43 43 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 18 18 18 29 29 29 F F F 18 360-379 25 25 25 39 39 39 F F F 19 380-399 24 24 24 38 38 38 F F F 20 400-419 24 24 24 38 38 38 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 25 25 25 39 39 39 F F F 23 460-479 24 24 24 38 38 38 F F F 24 480-499 26 26 26 42 42 42 F F F 25 500-519 25 25 25 39 39 39 F F F 26 520-539 27 27 27 43 43 43 F F F 27 540-559 24 24 24 38 38 38 F F F 28 560-579 18 18 18 29 29 29 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 27, Max flush: 43 Begin loop 14 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.32090319Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.343958297Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.344592288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.344670272Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.344692545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.344724639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"Crucible 07d650b6-ed86-46a1-b557-c9b52849f70f has session id: 7556879a-3d8f-411c-86e4-ca9264381fb7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.345914777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Upstairs opts: Upstairs UUID: 07d650b6-ed86-46a1-b557-c9b52849f70f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.345976078Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Crucible stats registered with UUID: 07d650b6-ed86-46a1-b557-c9b52849f70f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346003292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:45.346026548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346567475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346672603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.34670671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346796778Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346831096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"07d650b6-ed86-46a1-b557-c9b52849f70f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.3469449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.346989859Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347022884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347054126Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347084426Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347115908Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"ds_connection connected from Ok(127.0.0.1:61817)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347287376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"ds_connection connected from Ok(127.0.0.1:39728)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.347327124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.348021475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"ds_connection connected from Ok(127.0.0.1:47253)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.348451994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.349031718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.349126235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.349970801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.350020439Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.35012735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[0]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 39, 41, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351061424Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[0]R generation[0..12]: [25, 16, 26, 26, 27, 27, 25, 26, 24, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351146812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351202061Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[1]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 39, 41, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351250186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[1]R generation[0..12]: [25, 16, 26, 26, 27, 27, 25, 26, 24, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351301157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351339542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[2]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 39, 41, 38, 35, 16, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351374421Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[2]R generation[0..12]: [25, 16, 26, 26, 27, 27, 25, 26, 24, 22, 9, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351422135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351473577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351517293Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351563605Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Next flush: 44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351604315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351662369Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"":"downstairs","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.35170341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351731947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"0","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351769371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"1","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351820912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"client":"2","":"downstairs","session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351867194Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351904217Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"07d650b6-ed86-46a1-b557-c9b52849f70f is now active with session: 7556879a-3d8f-411c-86e4-ca9264381fb7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.351950187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.352025967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304,"session_id":"7556879a-3d8f-411c-86e4-ca9264381fb7"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.352072258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} {"msg":"Activated sub_volume 07d650b6-ed86-46a1-b557-c9b52849f70f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:45.352107598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 130 len 4096 data: 4 02/30 Write block 572 len 32768 data: 3 3 3 3 3 3 2 4 03/30 Read block 7 len 20480 04/30 Write block 126 len 4096 data: 5 05/30 Write block 166 len 12288 data: 4 4 3 06/30 Write block 525 len 16384 data: 5 4 4 4 07/30 Write block 358 len 40960 data: 2 3 3 3 3 3 3 3 2 2 08/30 Read block 375 len 8192 09/30 Read block 345 len 40960 10/30 Read block 211 len 32768 11/30 Write block 308 len 8192 data: 2 2 12/30 Write block 144 len 12288 data: 4 5 5 13/30 Flush 14/30 Write block 549 len 16384 data: 5 5 4 4 15/30 Flush 16/30 Flush 17/30 Flush 18/30 Write block 451 len 8192 data: 2 2 19/30 Read block 131 len 20480 20/30 Write block 289 len 28672 data: 5 3 3 3 2 2 3 21/30 Write block 405 len 4096 data: 5 22/30 Write block 463 len 12288 data: 4 3 4 23/30 Read block 98 len 12288 24/30 Write block 234 len 28672 data: 3 3 3 4 4 3 3 25/30 Read block 127 len 40960 26/30 Write block 390 len 4096 data: 2 27/30 Read block 196 len 32768 28/30 Read block 436 len 36864 29/30 Read block 357 len 16384 30/30 Write block 570 len 40960 data: 5 3 4 4 4 4 4 4 3 5 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 4 Done Done Done false 1006 Acked Write 10 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 24 26 Done 29 29 6 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:45.383404819Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1304} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 25 44 44 39 F F T <--- 7 140-159 28 28 26 44 44 41 F F F <--- 8 160-179 28 28 24 44 44 38 F F T <--- 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 18 18 18 29 29 29 T T F <--- 12 240-259 16 16 16 27 27 27 T T F <--- 13 260-279 27 27 27 43 43 43 F F F 14 280-299 27 27 27 43 43 43 T T F <--- 15 300-319 28 28 26 44 44 41 F F F <--- 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 18 44 44 29 F F F <--- 18 360-379 28 28 25 44 44 39 F F F <--- 19 380-399 24 24 24 38 38 38 T T F <--- 20 400-419 24 24 24 38 38 38 T T F <--- 21 420-439 25 25 25 39 39 39 F F F 22 440-459 25 25 25 39 39 39 T T F <--- 23 460-479 24 24 24 38 38 38 T T F <--- 24 480-499 26 26 26 42 42 42 F F F 25 500-519 25 25 25 39 39 39 F F F 26 520-539 28 28 27 44 44 43 F F T <--- 27 540-559 28 28 24 45 45 38 F F F <--- 28 560-579 28 28 18 44 44 29 T T T <--- 29 580-599 13 13 13 22 22 22 F F F Max gen: 28, Max flush: 45 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 28 44 44 44 F F F 7 140-159 28 28 28 44 44 44 F F F 8 160-179 28 28 28 44 44 44 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 29 29 29 46 46 46 F F F 13 260-279 27 27 27 43 43 43 F F F 14 280-299 29 29 29 46 46 46 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 28 44 44 44 F F F 18 360-379 28 28 28 44 44 44 F F F 19 380-399 29 29 29 46 46 46 F F F 20 400-419 29 29 29 46 46 46 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 29 29 29 46 46 46 F F F 23 460-479 29 29 29 46 46 46 F F F 24 480-499 26 26 26 42 42 42 F F F 25 500-519 25 25 25 39 39 39 F F F 26 520-539 28 28 28 44 44 44 F F F 27 540-559 28 28 28 45 45 45 F F F 28 560-579 29 29 29 46 46 46 F F F 29 580-599 13 13 13 22 22 22 F F F Max gen: 29, Max flush: 46 Begin loop 15 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.015161362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.03950962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.040017472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.040125085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.040148621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.040181897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"Crucible 766b3ffc-7273-4c7f-8c79-85ec7867d04b has session id: 5e76d8c8-8aa8-4254-b23a-eece51478dfc","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.041284893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Upstairs opts: Upstairs UUID: 766b3ffc-7273-4c7f-8c79-85ec7867d04b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.041348709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Crucible stats registered with UUID: 766b3ffc-7273-4c7f-8c79-85ec7867d04b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.041381975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:50.041405812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.041807304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04186087Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.041893675Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04242858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042514129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"766b3ffc-7273-4c7f-8c79-85ec7867d04b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042691128Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042719625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042745696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042771748Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042870693Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042901954Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.042929258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"ds_connection connected from Ok(127.0.0.1:45023)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04314789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"ds_connection connected from Ok(127.0.0.1:38278)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04340644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"ds_connection connected from Ok(127.0.0.1:35700)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.043567197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.045288304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.045386097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.045418731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.045966081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046347243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[0]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046500085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[0]R generation[0..12]: [25, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046569672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[1]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04659876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[1]R generation[0..12]: [25, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046627476Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046658137Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[2]R flush_numbers[0..12]: [39, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046687094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[2]R generation[0..12]: [25, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04671548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04674583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046776851Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046803053Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046827802Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.04685232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"":"downstairs","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046878161Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046901778Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"0","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046927779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"1","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046954632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"client":"2","":"downstairs","session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.046981064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.047005743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"766b3ffc-7273-4c7f-8c79-85ec7867d04b is now active with session: 5e76d8c8-8aa8-4254-b23a-eece51478dfc","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.047033558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.047057194Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312,"session_id":"5e76d8c8-8aa8-4254-b23a-eece51478dfc"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.047081663Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} {"msg":"Activated sub_volume 766b3ffc-7273-4c7f-8c79-85ec7867d04b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:50.047103225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 480 len 24576 data: 3 3 3 5 5 4 02/30 Read block 73 len 8192 03/30 Read block 567 len 40960 04/30 Write block 284 len 12288 data: 6 6 5 05/30 Read block 184 len 28672 06/30 Read block 431 len 8192 07/30 Read block 87 len 20480 08/30 Write block 519 len 28672 data: 5 6 6 6 5 6 6 09/30 Write block 525 len 8192 data: 7 5 10/30 Flush 11/30 Write block 380 len 8192 data: 6 5 12/30 Read block 349 len 28672 13/30 Read block 570 len 24576 14/30 Read block 448 len 12288 15/30 Read block 260 len 16384 16/30 Write block 504 len 24576 data: 5 3 3 3 3 4 17/30 Write block 280 len 16384 data: 8 8 8 6 18/30 Write block 7 len 40960 data: 2 2 3 4 6 6 6 4 4 3 19/30 Read block 130 len 32768 20/30 Write block 386 len 20480 data: 3 3 3 2 3 21/30 Read block 466 len 4096 22/30 Write block 259 len 40960 data: 8 5 4 4 3 3 2 2 3 2 23/30 Write block 587 len 16384 data: 3 3 3 3 24/30 Read block 30 len 16384 25/30 Read block 483 len 24576 26/30 Read block 278 len 28672 27/30 Read block 92 len 4096 28/30 Read block 581 len 12288 29/30 Write block 289 len 32768 data: 6 4 4 4 3 3 4 3 30/30 Write block 516 len 16384 data: 4 5 5 6 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1009)) None Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:50.07814918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1312} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 25 25 25 39 39 39 T F T <--- 1 020-039 16 16 16 27 27 27 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 28 44 44 44 F F F 7 140-159 28 28 28 44 44 44 F F F 8 160-179 28 28 28 44 44 44 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 29 29 29 46 46 46 T F T <--- 13 260-279 27 27 27 43 43 43 T F T <--- 14 280-299 30 29 30 47 46 47 T F T <--- 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 28 44 44 44 F F F 18 360-379 28 28 28 44 44 44 F F F 19 380-399 29 29 29 46 46 46 T F T <--- 20 400-419 29 29 29 46 46 46 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 29 29 29 46 46 46 F F F 23 460-479 29 29 29 46 46 46 F F F 24 480-499 30 26 30 47 42 47 F F F <--- 25 500-519 30 25 30 47 39 47 T F T <--- 26 520-539 30 28 30 47 44 47 F F F <--- 27 540-559 28 28 28 45 45 45 F F F 28 560-579 29 29 29 46 46 46 F F F 29 580-599 13 13 13 22 22 22 T F T <--- Max gen: 30, Max flush: 47 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 48 48 48 F F F 1 020-039 16 16 16 27 27 27 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 28 44 44 44 F F F 7 140-159 28 28 28 44 44 44 F F F 8 160-179 28 28 28 44 44 44 F F F 9 180-199 22 22 22 35 35 35 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 31 31 31 48 48 48 F F F 13 260-279 31 31 31 48 48 48 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 28 44 44 44 F F F 18 360-379 28 28 28 44 44 44 F F F 19 380-399 31 31 31 48 48 48 F F F 20 400-419 29 29 29 46 46 46 F F F 21 420-439 25 25 25 39 39 39 F F F 22 440-459 29 29 29 46 46 46 F F F 23 460-479 29 29 29 46 46 46 F F F 24 480-499 30 30 30 47 47 47 F F F 25 500-519 31 31 31 48 48 48 F F F 26 520-539 30 30 30 47 47 47 F F F 27 540-559 28 28 28 45 45 45 F F F 28 560-579 29 29 29 46 46 46 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 31, Max flush: 48 Begin loop 16 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.660614816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.684770785Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.685502779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.685620932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.685670951Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.685725348Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"Crucible f1c1aa97-1657-4ac4-8c50-a5585764cdc0 has session id: 3dd79138-67ba-4248-9f5f-b73d786772b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.686990355Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Upstairs opts: Upstairs UUID: f1c1aa97-1657-4ac4-8c50-a5585764cdc0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.687123968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Crucible stats registered with UUID: f1c1aa97-1657-4ac4-8c50-a5585764cdc0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.687201632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:54.687264606Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.687603044Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.687709915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.68778847Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.68819395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688264028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"f1c1aa97-1657-4ac4-8c50-a5585764cdc0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688315389Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688372753Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.68843713Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688518861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688583609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.6886499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.6887159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:62337)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.688787732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:64870)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.689585255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:35744)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.689662057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.691397171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.691494764Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.691528831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692122372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.69238473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[0]R flush_numbers[0..12]: [48, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692464277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[0]R generation[0..12]: [31, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692507312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692549966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[1]R flush_numbers[0..12]: [48, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692598522Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[1]R generation[0..12]: [31, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692640585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692681205Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[2]R flush_numbers[0..12]: [48, 27, 41, 41, 43, 43, 44, 44, 44, 35, 16, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692718679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[2]R generation[0..12]: [31, 16, 26, 26, 27, 27, 28, 28, 28, 22, 9, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692754409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692791913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692825099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692857863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Next flush: 49","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692890438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692923653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"":"downstairs","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692956238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.692986297Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"0","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693021887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"1","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.69305392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"client":"2","":"downstairs","session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693088789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693127365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"f1c1aa97-1657-4ac4-8c50-a5585764cdc0 is now active with session: 3dd79138-67ba-4248-9f5f-b73d786772b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693164629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693196091Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320,"session_id":"3dd79138-67ba-4248-9f5f-b73d786772b3"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693227894Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} {"msg":"Wait for a query_work_queue command to finish before sending IO Activated sub_volume f1c1aa97-1657-4ac4-8c50-a5585764cdc0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:54.693275658Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 261 len 16384 data: 5 5 4 4 02/30 Write block 21 len 36864 data: 4 2 2 2 2 2 3 3 4 03/30 Read block 38 len 36864 04/30 Read block 550 len 24576 05/30 Write block 272 len 20480 data: 5 5 3 4 5 06/30 Read block 252 len 12288 07/30 Read block 491 len 12288 08/30 Write block 259 len 4096 data: 9 09/30 Read block 182 len 20480 10/30 Write block 212 len 12288 data: 2 2 2 11/30 Read block 428 len 28672 12/30 Write block 361 len 20480 data: 4 4 4 4 4 13/30 Write block 499 len 12288 data: 4 5 5 14/30 Read block 235 len 36864 15/30 Read block 292 len 16384 16/30 Write block 427 len 40960 data: 4 4 4 5 5 4 4 5 5 5 17/30 Write block 182 len 40960 data: 4 6 4 4 4 4 3 2 2 2 18/30 Read block 289 len 24576 19/30 Write block 525 len 24576 data: 8 6 5 5 3 3 20/30 Read block 329 len 32768 21/30 Write block 455 len 4096 data: 2 22/30 Write block 68 len 4096 data: 5 23/30 Write block 401 len 32768 data: 4 5 6 5 6 5 4 3 24/30 Write block 476 len 4096 data: 3 25/30 Write block 509 len 12288 data: 5 4 4 26/30 Write block 437 len 36864 data: 5 3 3 3 4 4 4 4 5 27/30 Write block 565 len 28672 data: 4 4 4 4 3 6 4 28/30 Write block 392 len 40960 data: 2 2 2 2 2 2 3 4 4 5 29/30 Read block 109 len 4096 30/30 Read block 83 len 12288 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 3 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Read 5 Done Done Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 21 0 21 Done 30 9 30 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:54.725008428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1320} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 48 48 48 F F F 1 020-039 16 16 16 27 27 27 T T T 2 040-059 26 26 26 41 41 41 F F F 3 060-079 26 26 26 41 41 41 T F T <--- 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 28 44 44 44 F F F 7 140-159 28 28 28 44 44 44 F F F 8 160-179 28 28 28 44 44 44 F F F 9 180-199 22 22 22 35 35 35 T F T <--- 10 200-219 9 9 9 16 16 16 T F T <--- 11 220-239 29 29 29 46 46 46 F F F 12 240-259 31 31 31 48 48 48 T T T 13 260-279 31 31 31 48 48 48 T T T 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 28 44 44 44 F F F 18 360-379 28 28 28 44 44 44 T F T <--- 19 380-399 31 31 31 48 48 48 T F T <--- 20 400-419 29 29 29 46 46 46 T F T <--- 21 420-439 25 25 25 39 39 39 T F T <--- 22 440-459 29 29 29 46 46 46 T F T <--- 23 460-479 29 29 29 46 46 46 T F T <--- 24 480-499 30 30 30 47 47 47 T F T <--- 25 500-519 31 31 31 48 48 48 T F T <--- 26 520-539 30 30 30 47 47 47 T F T <--- 27 540-559 28 28 28 45 45 45 F F F 28 560-579 29 29 29 46 46 46 T F T <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 31, Max flush: 48 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 48 48 48 F F F 1 020-039 32 32 32 49 49 49 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 32 32 32 49 49 49 F F F 4 080-099 27 27 27 43 43 43 F F F 5 100-119 27 27 27 43 43 43 F F F 6 120-139 28 28 28 44 44 44 F F F 7 140-159 28 28 28 44 44 44 F F F 8 160-179 28 28 28 44 44 44 F F F 9 180-199 32 32 32 49 49 49 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 28 28 44 44 44 F F F 18 360-379 32 32 32 49 49 49 F F F 19 380-399 32 32 32 49 49 49 F F F 20 400-419 32 32 32 49 49 49 F F F 21 420-439 32 32 32 49 49 49 F F F 22 440-459 32 32 32 49 49 49 F F F 23 460-479 32 32 32 49 49 49 F F F 24 480-499 32 32 32 49 49 49 F F F 25 500-519 32 32 32 49 49 49 F F F 26 520-539 32 32 32 49 49 49 F F F 27 540-559 28 28 28 45 45 45 F F F 28 560-579 32 32 32 49 49 49 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 32, Max flush: 49 Begin loop 17 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.361830159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:59.383852537Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.383967604Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.402172469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.40279636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.4029635Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.403008138Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.403075571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} {"msg":"Crucible 5b72c27c-cc26-4d39-8d9c-29c21418eae9 has session id: 37d9061e-2e6c-44e2-ba51-8f8766da70ec","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.403857985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Upstairs opts: Upstairs UUID: 5b72c27c-cc26-4d39-8d9c-29c21418eae9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.403927953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Crucible stats registered with UUID: 5b72c27c-cc26-4d39-8d9c-29c21418eae9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.403973032Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:59.404005316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.40434182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.404459432Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.404497607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.404841596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.404884761Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} {"msg":"5b72c27c-cc26-4d39-8d9c-29c21418eae9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.404998646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405034687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405069456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405103352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405136598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405264711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405303107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:57943)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405641564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:48323)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405722614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:54157)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.405768665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.408035499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.408241184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.408433183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.408940795Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409057876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[0]R flush_numbers[0..12]: [48, 49, 41, 49, 43, 43, 44, 44, 44, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.40922715Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[0]R generation[0..12]: [31, 32, 26, 32, 27, 27, 28, 28, 28, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409268382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409308962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[1]R flush_numbers[0..12]: [48, 49, 41, 49, 43, 43, 44, 44, 44, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409346476Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[1]R generation[0..12]: [31, 32, 26, 32, 27, 27, 28, 28, 28, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.40938392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409423888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[2]R flush_numbers[0..12]: [48, 49, 41, 49, 43, 43, 44, 44, 44, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409461603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[2]R generation[0..12]: [31, 32, 26, 32, 27, 27, 28, 28, 28, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409498866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409545268Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Max found gen is 33","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:00:59.409578233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Generation requested: 34 >= found:33","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409621448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Next flush: 50","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409653962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409686396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"":"downstairs","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409720764Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409751856Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"0","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409786083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"1","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409819699Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"client":"2","":"downstairs","session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409854348Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409886732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"5b72c27c-cc26-4d39-8d9c-29c21418eae9 is now active with session: 37d9061e-2e6c-44e2-ba51-8f8766da70ec","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409923073Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409953674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328,"session_id":"37d9061e-2e6c-44e2-ba51-8f8766da70ec"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.409985777Z","hostname":"bmat-EVT22200007-0000ec3a","pid"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :1328} {"msg":"Activated sub_volume 5b72c27c-cc26-4d39-8d9c-29c21418eae9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:00:59.41002758Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 529 len 24576 data: 4 4 5 6 7 5 02/30 Flush 03/30 Write block 352 len 28672 data: 3 3 3 3 3 2 3 04/30 Flush 05/30 Write block 166 len 24576 data: 5 5 4 4 4 4 06/30 Read block 578 len 20480 07/30 Read block 253 len 20480 08/30 Write block 377 len 24576 data: 4 6 7 7 6 5 09/30 Read block 221 len 36864 10/30 Write block 413 len 20480 data: 4 5 4 4 4 11/30 Flush 12/30 Write block 448 len 24576 data: 5 4 4 3 3 2 13/30 Write block 150 len 20480 data: 4 5 3 3 3 14/30 Read block 426 len 24576 15/30 Write block 68 len 40960 data: 6 3 2 2 3 5 5 7 9 9 16/30 Write block 88 len 20480 data: 6 6 4 3 3 17/30 Read block 411 len 8192 18/30 Read block 470 len 28672 19/30 Write block 495 len 20480 data: 3 4 4 4 5 20/30 Write block 140 len 24576 data: 4 4 4 4 5 6 21/30 Read block 188 len 16384 22/30 Read block 301 len 12288 23/30 Read block 47 len 12288 24/30 Read block 9 len 40960 25/30 Write block 130 len 16384 data: 5 4 4 4 26/30 Write block 346 len 32768 data: 3 3 3 2 3 3 4 4 27/30 Read block 2 len 32768 28/30 Read block 56 len 36864 29/30 Write block 116 len 36864 data: 6 5 5 2 4 4 3 3 3 30/30 Read block 386 len 36864 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 7 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 0 29 Done 0 27 28 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:00:59.445374732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1328} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 48 48 48 F F F 1 020-039 32 32 32 49 49 49 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 32 32 32 49 49 49 F T T <--- 4 080-099 27 27 27 43 43 43 F T T <--- 5 100-119 27 27 27 43 43 43 F T T <--- 6 120-139 28 28 28 44 44 44 F T T <--- 7 140-159 28 28 28 44 44 44 F T T <--- 8 160-179 28 34 34 44 52 52 F F F <--- 9 180-199 32 32 32 49 49 49 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 28 34 34 44 51 51 F T T <--- 18 360-379 32 34 34 49 52 52 F F F <--- 19 380-399 32 34 34 49 52 52 F F F <--- 20 400-419 32 34 34 49 52 52 F F F <--- 21 420-439 32 32 32 49 49 49 F F F 22 440-459 32 32 32 49 49 49 F T T <--- 23 460-479 32 32 32 49 49 49 F F F 24 480-499 32 32 32 49 49 49 F T T <--- 25 500-519 32 32 32 49 49 49 F F F 26 520-539 34 34 34 50 50 50 F F F 27 540-559 28 28 28 45 45 45 F F F 28 560-579 32 32 32 49 49 49 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 34, Max flush: 52 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 48 48 48 F F F 1 020-039 32 32 32 49 49 49 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 35 35 35 53 53 53 F F F 4 080-099 35 35 35 53 53 53 F F F 5 100-119 35 35 35 53 53 53 F F F 6 120-139 35 35 35 53 53 53 F F F 7 140-159 35 35 35 53 53 53 F F F 8 160-179 34 34 34 52 52 52 F F F 9 180-199 32 32 32 49 49 49 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 28 28 44 44 44 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 34 34 34 52 52 52 F F F 19 380-399 34 34 34 52 52 52 F F F 20 400-419 34 34 34 52 52 52 F F F 21 420-439 32 32 32 49 49 49 F F F 22 440-459 35 35 35 53 53 53 F F F 23 460-479 32 32 32 49 49 49 F F F 24 480-499 35 35 35 53 53 53 F F F 25 500-519 32 32 32 49 49 49 F F F 26 520-539 34 34 34 50 50 50 F F F 27 540-559 28 28 28 45 45 45 F F F 28 560-579 32 32 32 49 49 49 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 35, Max flush: 53 Begin loop 18 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.049923295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:04.0694711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.069555567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.08775467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.088254577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.088328402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.088352079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.088386527Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"Crucible 2facd7f1-d3db-4465-b5fb-2a7183ce0c77 has session id: 7ca1e9b1-8506-426c-938f-565202b0576b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.089324107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Upstairs opts: Upstairs UUID: 2facd7f1-d3db-4465-b5fb-2a7183ce0c77, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.089506177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Crucible stats registered with UUID: 2facd7f1-d3db-4465-b5fb-2a7183ce0c77","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.089546827Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:04.089628508Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090155909Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090244173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090270585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090661927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090692267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"2facd7f1-d3db-4465-b5fb-2a7183ce0c77 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090807775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090837564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.09086083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090883224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090905298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.090928644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.091052278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"ds_connection connected from Ok(127.0.0.1:61818)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.091081756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"ds_connection connected from Ok(127.0.0.1:46342)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.091106565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"ds_connection connected from Ok(127.0.0.1:39496)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.091259486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.093321896Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.093602099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.094029703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.094131374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.094622814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[0]R flush_numbers[0..12]: [48, 49, 41, 53, 53, 53, 53, 53, 52, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095176075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[0]R generation[0..12]: [31, 32, 26, 35, 35, 35, 35, 35, 34, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095227647Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.09526949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[1]R flush_numbers[0..12]: [48, 49, 41, 53, 53, 53, 53, 53, 52, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095307775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[1]R generation[0..12]: [31, 32, 26, 35, 35, 35, 35, 35, 34, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.09534586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.09538601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[2]R flush_numbers[0..12]: [48, 49, 41, 53, 53, 53, 53, 53, 52, 49, 49, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095423854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[2]R generation[0..12]: [31, 32, 26, 35, 35, 35, 35, 35, 34, 32, 32, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095461759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095502209Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095535254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095570013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Next flush: 54","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095602767Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095635322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095678847Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.09571038Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"0","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095749657Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"1","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095784426Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"client":"2","":"downstairs","session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095820257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095852961Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"2facd7f1-d3db-4465-b5fb-2a7183ce0c77 is now active with session: 7ca1e9b1-8506-426c-938f-565202b0576b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095890134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095921106Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336,"session_id":"7ca1e9b1-8506-426c-938f-565202b0576b"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:01:04.09595372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} {"msg":"Activated sub_volume 2facd7f1-d3db-4465-b5fb-2a7183ce0c77","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:04.095987988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 181 len 40960 data: 4 5 7 5 5 5 5 4 3 3 02/30 Read block 503 len 40960 03/30 Write block 124 len 8192 data: 4 4 04/30 Read block 246 len 40960 05/30 Read block 218 len 12288 06/30 Write block 165 len 28672 data: 3 6 6 5 5 5 5 07/30 Flush 08/30 Write block 301 len 12288 data: 4 4 4 09/30 Read block 187 len 20480 10/30 Read block 41 len 24576 11/30 Read block 486 len 24576 12/30 Write block 11 len 24576 data: 7 7 7 5 5 4 13/30 Write block 74 len 36864 data: 6 8 10 10 9 8 8 10 10 14/30 Flush 15/30 Write block 507 len 4096 data: 4 16/30 Read block 191 len 40960 17/30 Read block 274 len 32768 18/30 Read block 36 len 36864 19/30 Read block 50 len 24576 20/30 Read block 181 len 12288 21/30 Read block 124 len 20480 22/30 Write block 366 len 8192 data: 3 3 23/30 Read block 65 len 28672 24/30 Write block 476 len 4096 data: 4 25/30 Read block 88 len 32768 26/30 Write block 48 len 40960 data: 3 2 3 3 2 2 4 4 4 4 27/30 Read block 322 len 4096 28/30 Write block 542 len 12288 data: 3 2 3 29/30 Write block 191 len 40960 data: 3 2 2 3 3 4 4 3 2 3 30/30 Write block 573 len 20480 data: 5 5 5 5 5 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 3 Done Done Done false 1008 Acked Read 5 Done Done Done false 1009 Acked Read 6 Done Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 3 Sent Sent Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 3 2 25 Done 3 20 21 44 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:04.126908645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1336} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 36 36 48 55 55 F F F <--- 1 020-039 32 32 32 49 49 49 F F F 2 040-059 26 26 26 41 41 41 F T T <--- 3 060-079 35 36 36 53 55 55 F F F <--- 4 080-099 35 36 36 53 55 55 F F F <--- 5 100-119 35 35 35 53 53 53 F F F 6 120-139 36 36 36 54 54 54 F F F 7 140-159 35 35 35 53 53 53 F F F 8 160-179 36 36 36 54 54 54 F F F 9 180-199 36 36 36 54 54 54 F T T <--- 10 200-219 32 32 32 49 49 49 F T T <--- 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 28 36 36 44 55 55 T F F <--- 16 320-339 20 20 20 30 30 30 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 34 34 34 52 52 52 F T T <--- 19 380-399 34 34 34 52 52 52 F F F 20 400-419 34 34 34 52 52 52 F F F 21 420-439 32 32 32 49 49 49 F F F 22 440-459 35 35 35 53 53 53 F F F 23 460-479 32 32 32 49 49 49 F T T <--- 24 480-499 35 35 35 53 53 53 F F F 25 500-519 32 32 32 49 49 49 F T T <--- 26 520-539 34 34 34 50 50 50 F F F 27 540-559 28 28 28 45 45 45 F T T <--- 28 560-579 32 32 32 49 49 49 F T T <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 36, Max flush: 55 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 55 55 55 F F F 1 020-039 32 32 32 49 49 49 F F F 2 040-059 37 37 37 56 56 56 F F F 3 060-079 36 36 36 55 55 55 F F F 4 080-099 36 36 36 55 55 55 F F F 5 100-119 35 35 35 53 53 53 F F F 6 120-139 36 36 36 54 54 54 F F F 7 140-159 35 35 35 53 53 53 F F F 8 160-179 36 36 36 54 54 54 F F F 9 180-199 37 37 37 56 56 56 F F F 10 200-219 37 37 37 56 56 56 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 36 36 36 55 55 55 F F F 16 320-339 20 20 20 30 30 30 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 37 37 37 56 56 56 F F F 19 380-399 34 34 34 52 52 52 F F F 20 400-419 34 34 34 52 52 52 F F F 21 420-439 32 32 32 49 49 49 F F F 22 440-459 35 35 35 53 53 53 F F F 23 460-479 37 37 37 56 56 56 F F F 24 480-499 35 35 35 53 53 53 F F F 25 500-519 37 37 37 56 56 56 F F F 26 520-539 34 34 34 50 50 50 F F F 27 540-559 37 37 37 56 56 56 F F F 28 560-579 37 37 37 56 56 56 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 37, Max flush: 56 Begin loop 19 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.686105504Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.710865404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.711688358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.711781312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.711814126Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.711864997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"Crucible 4886206d-b1f4-46ca-9342-3e9a250864f5 has session id: 0543c65e-6b63-4b2d-9700-7eeb7ec841cf","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.713128811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Upstairs opts: Upstairs UUID: 4886206d-b1f4-46ca-9342-3e9a250864f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.713202987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Crucible stats registered with UUID: 4886206d-b1f4-46ca-9342-3e9a250864f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.71323974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:08.713278085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.71369112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.713750538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.713786238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714024739Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714065419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"4886206d-b1f4-46ca-9342-3e9a250864f5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714179043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714214102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714249242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.7142838Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714317296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714353618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:43577)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714464096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714501289Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:49654)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.71480089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:36480)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.714848915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.716689076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.716800296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.71721814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.717443164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.717568612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[0]R flush_numbers[0..12]: [55, 49, 56, 55, 55, 53, 54, 53, 54, 56, 56, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718588585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[0]R generation[0..12]: [36, 32, 37, 36, 36, 35, 36, 35, 36, 37, 37, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718679524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718718261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[1]R flush_numbers[0..12]: [55, 49, 56, 55, 55, 53, 54, 53, 54, 56, 56, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.71875347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[1]R generation[0..12]: [36, 32, 37, 36, 36, 35, 36, 35, 36, 37, 37, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718787487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718823909Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[2]R flush_numbers[0..12]: [55, 49, 56, 55, 55, 53, 54, 53, 54, 56, 56, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718857876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[2]R generation[0..12]: [36, 32, 37, 36, 36, 35, 36, 35, 36, 37, 37, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718891302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718927333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.718975899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719036198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Next flush: 57","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719066718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719096287Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"":"downstairs","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719127358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719155373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"0","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719186505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"1","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719217085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"client":"2","":"downstairs","session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719248397Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719277564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"4886206d-b1f4-46ca-9342-3e9a250864f5 is now active with session: 0543c65e-6b63-4b2d-9700-7eeb7ec841cf","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719322653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719351019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344,"session_id":"0543c65e-6b63-4b2d-9700-7eeb7ec841cf"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719380357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} {"msg":"Activated sub_volume 4886206d-b1f4-46ca-9342-3e9a250864f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:08.719406609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 43 len 16384 data: 3 3 3 3 02/30 Write block 61 len 16384 data: 5 4 5 5 03/30 Write block 434 len 20480 data: 6 6 6 6 4 04/30 Write block 44 len 28672 data: 4 4 4 3 4 3 4 05/30 Read block 498 len 40960 06/30 Read block 235 len 20480 07/30 Write block 550 len 32768 data: 6 5 5 3 4 4 4 3 08/30 Write block 191 len 32768 data: 4 3 3 4 4 5 5 4 09/30 Read block 196 len 28672 10/30 Read block 358 len 40960 11/30 Write block 53 len 12288 data: 3 5 5 12/30 Read block 434 len 12288 13/30 Flush 14/30 Write block 314 len 40960 data: 3 4 2 2 2 3 3 3 2 2 15/30 Read block 127 len 32768 16/30 Read block 248 len 24576 17/30 Flush 18/30 Write block 260 len 28672 data: 6 6 6 5 5 3 3 19/30 Read block 258 len 16384 20/30 Write block 99 len 32768 data: 3 3 3 3 3 3 2 2 21/30 Write block 5 len 8192 data: 2 2 22/30 Write block 206 len 12288 data: 4 3 2 23/30 Read block 43 len 28672 24/30 Read block 66 len 4096 25/30 Write block 144 len 24576 data: 6 7 6 5 5 4 26/30 Write block 65 len 40960 data: 5 5 5 7 4 3 3 4 6 7 27/30 Write block 478 len 8192 data: 3 3 28/30 Write block 384 len 4096 data: 4 29/30 Write block 403 len 4096 data: 7 30/30 Write block 113 len 32768 data: 5 3 5 7 6 6 3 5 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Write 1 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:08.748861835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1344} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 55 55 55 F T T <--- 1 020-039 32 32 32 49 49 49 F F F 2 040-059 37 38 38 56 57 57 F F F <--- 3 060-079 36 38 38 55 57 57 F T T <--- 4 080-099 36 36 36 55 55 55 F T T <--- 5 100-119 35 35 35 53 53 53 F T T <--- 6 120-139 36 36 36 54 54 54 F T T <--- 7 140-159 35 35 35 53 53 53 F T T <--- 8 160-179 36 36 36 54 54 54 F F F 9 180-199 37 38 38 56 57 57 F F F <--- 10 200-219 37 37 37 56 56 56 F T T <--- 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 32 32 32 49 49 49 F T T <--- 14 280-299 31 31 31 48 48 48 F F F 15 300-319 36 38 38 55 58 58 F F F <--- 16 320-339 20 38 38 30 58 58 F F F <--- 17 340-359 35 35 35 53 53 53 F F F 18 360-379 37 37 37 56 56 56 F F F 19 380-399 34 34 34 52 52 52 F T T <--- 20 400-419 34 34 34 52 52 52 F T T <--- 21 420-439 32 38 38 49 57 57 F F F <--- 22 440-459 35 35 35 53 53 53 F F F 23 460-479 37 37 37 56 56 56 F T T <--- 24 480-499 35 35 35 53 53 53 F F F 25 500-519 37 37 37 56 56 56 F F F 26 520-539 34 34 34 50 50 50 F F F 27 540-559 37 38 38 56 57 57 F F F <--- 28 560-579 37 37 37 56 56 56 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 38, Max flush: 58 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 32 32 32 49 49 49 F F F 2 040-059 38 38 38 57 57 57 F F F 3 060-079 39 39 39 59 59 59 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 39 39 59 59 59 F F F 7 140-159 39 39 39 59 59 59 F F F 8 160-179 36 36 36 54 54 54 F F F 9 180-199 38 38 38 57 57 57 F F F 10 200-219 39 39 39 59 59 59 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 31 31 31 48 48 48 F F F 15 300-319 38 38 38 58 58 58 F F F 16 320-339 38 38 38 58 58 58 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 37 37 37 56 56 56 F F F 19 380-399 39 39 39 59 59 59 F F F 20 400-419 39 39 39 59 59 59 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 35 35 35 53 53 53 F F F 23 460-479 39 39 39 59 59 59 F F F 24 480-499 35 35 35 53 53 53 F F F 25 500-519 37 37 37 56 56 56 F F F 26 520-539 34 34 34 50 50 50 F F F 27 540-559 38 38 38 57 57 57 F F F 28 560-579 37 37 37 56 56 56 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 39, Max flush: 59 Begin loop 20 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.413312207Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:13.434350613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.434443556Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.451874163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.452527162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.452605797Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.452633191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.452673661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"Crucible 09583290-8f71-4be8-85c8-73700665187d has session id: 88270e63-6c95-4209-85cd-d7a3a1e69351","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.453886444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Upstairs opts: Upstairs UUID: 09583290-8f71-4be8-85c8-73700665187d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454020649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Crucible stats registered with UUID: 09583290-8f71-4be8-85c8-73700665187d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454068594Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:13.45410722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454413484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454492641Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454535716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454693457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454736602Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"09583290-8f71-4be8-85c8-73700665187d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454845337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.45489767Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.454938711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.4549787Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455017516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455058317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455098626Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"ds_connection connected from Ok(127.0.0.1:41454)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455162583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"ds_connection connected from Ok(127.0.0.1:61006)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455206339Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"ds_connection connected from Ok(127.0.0.1:39250)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.455431794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.456825464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.456961112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.457469685Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.457777703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.457993118Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[0]R flush_numbers[0..12]: [59, 49, 57, 59, 59, 59, 59, 59, 54, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459023672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[0]R generation[0..12]: [39, 32, 38, 39, 39, 39, 39, 39, 36, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459053902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459086686Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[1]R flush_numbers[0..12]: [59, 49, 57, 59, 59, 59, 59, 59, 54, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459110163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[1]R generation[0..12]: [39, 32, 38, 39, 39, 39, 39, 39, 36, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459133308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459157546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[2]R flush_numbers[0..12]: [59, 49, 57, 59, 59, 59, 59, 59, 54, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459180371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[2]R generation[0..12]: [39, 32, 38, 39, 39, 39, 39, 39, 36, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459203346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459227244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459247464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459268435Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459288134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459307843Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"":"downstairs","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459328624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459347922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"0","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459368783Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"1","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459393361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"client":"2","":"downstairs","session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459415114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459435214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"09583290-8f71-4be8-85c8-73700665187d is now active with session: 88270e63-6c95-4209-85cd-d7a3a1e69351","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459457548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459476395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352,"session_id":"88270e63-6c95-4209-85cd-d7a3a1e69351"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.459496184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} {"msg":"Activated sub_volume 09583290-8f71-4be8-85c8-73700665187d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:13.45951412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 375 len 28672 data: 3 3 5 7 8 8 7 02/30 Read block 263 len 40960 03/30 Read block 88 len 36864 04/30 Read block 549 len 4096 05/30 Write block 536 len 40960 data: 6 6 5 4 3 3 4 3 4 3 06/30 Write block 311 len 24576 data: 3 4 4 4 5 3 07/30 Read block 185 len 8192 08/30 Read block 29 len 12288 09/30 Read block 220 len 4096 10/30 Read block 473 len 36864 11/30 Write block 547 len 12288 data: 4 4 6 12/30 Read block 205 len 28672 13/30 Write block 466 len 12288 data: 3 3 4 14/30 Read block 496 len 28672 15/30 Read block 120 len 20480 16/30 Write block 441 len 24576 data: 5 5 5 5 6 6 17/30 Flush 18/30 Write block 284 len 4096 data: 7 19/30 Flush 20/30 Write block 497 len 24576 data: 5 5 6 6 6 5 21/30 Write block 383 len 36864 data: 4 5 4 4 4 4 3 4 2 22/30 Read block 155 len 4096 23/30 Write block 27 len 16384 data: 4 4 5 4 24/30 Read block 391 len 24576 25/30 Read block 578 len 36864 26/30 Write block 169 len 32768 data: 6 6 6 4 3 4 4 3 27/30 Read block 133 len 4096 28/30 Read block 522 len 28672 29/30 Write block 60 len 28672 data: 4 6 5 6 6 6 6 30/30 Write block 480 len 12288 data: 4 4 4 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 2 2 31 Done 3 28 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:13.49229586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1352} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 32 32 32 49 49 49 F T T <--- 2 040-059 38 38 38 57 57 57 F F F 3 060-079 39 39 39 59 59 59 F T T <--- 4 080-099 39 39 39 59 59 59 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 39 39 59 59 59 F F F 7 140-159 39 39 39 59 59 59 F F F 8 160-179 36 36 36 54 54 54 F T T <--- 9 180-199 38 38 38 57 57 57 F F F 10 200-219 39 39 39 59 59 59 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 31 40 40 48 61 61 F F F <--- 15 300-319 38 40 40 58 60 60 F F F <--- 16 320-339 38 38 38 58 58 58 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 37 40 40 56 60 60 T F F <--- 19 380-399 39 40 40 59 60 60 T T T <--- 20 400-419 39 39 39 59 59 59 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 35 40 40 53 60 60 F F F <--- 23 460-479 39 40 40 59 60 60 F F F <--- 24 480-499 35 35 35 53 53 53 F T T <--- 25 500-519 37 37 37 56 56 56 F T T <--- 26 520-539 34 40 40 50 60 60 F F F <--- 27 540-559 38 40 40 57 60 60 F F F <--- 28 560-579 37 37 37 56 56 56 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 40, Max flush: 61 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 41 41 41 62 62 62 F F F 2 040-059 38 38 38 57 57 57 F F F 3 060-079 41 41 41 62 62 62 F F F 4 080-099 39 39 39 59 59 59 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 39 39 59 59 59 F F F 7 140-159 39 39 39 59 59 59 F F F 8 160-179 41 41 41 62 62 62 F F F 9 180-199 38 38 38 57 57 57 F F F 10 200-219 39 39 39 59 59 59 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 61 61 61 F F F 15 300-319 40 40 40 60 60 60 F F F 16 320-339 38 38 38 58 58 58 F F F 17 340-359 35 35 35 53 53 53 F F F 18 360-379 40 40 40 60 60 60 F F F 19 380-399 41 41 41 62 62 62 F F F 20 400-419 39 39 39 59 59 59 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 40 40 40 60 60 60 F F F 23 460-479 40 40 40 60 60 60 F F F 24 480-499 41 41 41 62 62 62 F F F 25 500-519 41 41 41 62 62 62 F F F 26 520-539 40 40 40 60 60 60 F F F 27 540-559 40 40 40 60 60 60 F F F 28 560-579 37 37 37 56 56 56 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 41, Max flush: 62 Begin loop 21 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.074577061Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:18.093739306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.093847048Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.110107536Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.110563406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.110647932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.11067223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.110709193Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"Crucible 5923ec33-518b-46f9-bc62-dbdc996b3d89 has session id: 483ba13e-a993-49e0-b873-a0ded9f67fb4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.111799375Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Upstairs opts: Upstairs UUID: 5923ec33-518b-46f9-bc62-dbdc996b3d89, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.111914632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Crucible stats registered with UUID: 5923ec33-518b-46f9-bc62-dbdc996b3d89","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.111952667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:18.111978959Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.112282688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.112352256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.112392545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.112710092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.112758718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"5923ec33-518b-46f9-bc62-dbdc996b3d89 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113135792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113239506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113274966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113306208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113336227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.11336806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113398781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:42729)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113431235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:57001)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113511613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:63638)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.113739032Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.115447373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.115528393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.115584985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116368541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116412217Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[0]R flush_numbers[0..12]: [59, 62, 57, 62, 59, 59, 59, 59, 62, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116467907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[0]R generation[0..12]: [39, 41, 38, 41, 39, 39, 39, 39, 41, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116500522Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116620549Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[1]R flush_numbers[0..12]: [59, 62, 57, 62, 59, 59, 59, 59, 62, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116724233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[1]R generation[0..12]: [39, 41, 38, 41, 39, 39, 39, 39, 41, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116765755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} Wait for a query_work_queue command to finish before sending IO {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116809812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[2]R flush_numbers[0..12]: [59, 62, 57, 62, 59, 59, 59, 59, 62, 57, 59, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116841675Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[2]R generation[0..12]: [39, 41, 38, 41, 39, 39, 39, 39, 41, 38, 39, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.11687986Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116913316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.11694039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116968435Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Next flush: 63","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.116994907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117022381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"":"downstairs","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117050417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117076067Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"0","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117104133Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"1","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","levelDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":30,"time":"2026-03-06T06:01:18.117141877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"client":"2","":"downstairs","session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117171897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.11719898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"5923ec33-518b-46f9-bc62-dbdc996b3d89 is now active with session: 483ba13e-a993-49e0-b873-a0ded9f67fb4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117229109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117254409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360,"session_id":"483ba13e-a993-49e0-b873-a0ded9f67fb4"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117281002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} {"msg":"Activated sub_volume 5923ec33-518b-46f9-bc62-dbdc996b3d89","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:18.117304348Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 532 len 32768 data: 7 8 6 5 7 7 6 5 02/30 Read block 568 len 4096 03/30 Write block 55 len 24576 data: 6 5 5 4 4 5 04/30 Write block 539 len 20480 data: 6 4 4 5 4 05/30 Read block 108 len 32768 06/30 Flush 07/30 Flush 08/30 Write block 399 len 20480 data: 5 5 6 6 8 09/30 Read block 49 len 20480 10/30 Write block 563 len 28672 data: 4 4 5 5 5 5 4 11/30 Write block 62 len 36864 data: 6 7 7 7 7 6 8 5 4 12/30 Read block 346 len 28672 13/30 Read block 79 len 12288 14/30 Write block 402 len 16384 data: 7 9 6 7 15/30 Read block 234 len 40960 16/30 Read block 205 len 28672 17/30 Write block 84 len 40960 data: 7 9 7 6 7 7 5 4 4 3 18/30 Write block 146 len 4096 data: 7 19/30 Read block 411 len 20480 20/30 Write block 393 len 4096 data: 3 21/30 Write block 194 len 24576 data: 5 5 6 6 5 3 22/30 Read block 158 len 12288 23/30 Write block 248 len 4096 data: 2 24/30 Write block 357 len 16384 data: 3 4 4 4 25/30 Write block 311 len 32768 data: 4 5 5 5 6 4 3 3 26/30 Read block 67 len 12288 27/30 Write block 202 len 36864 data: 4 4 4 4 5 4 3 2 2 28/30 Read block 305 len 16384 29/30 Read block 306 len 12288 30/30 Write block 339 len 36864 data: 2 2 2 2 2 2 3 4 4 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:18.145959475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1360} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 41 41 41 62 62 62 F F F 2 040-059 38 42 42 57 63 63 T F F <--- 3 060-079 41 42 42 62 63 63 T T T <--- 4 080-099 39 39 39 59 59 59 F T T <--- 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 39 39 59 59 59 F F F 7 140-159 39 39 39 59 59 59 F T T <--- 8 160-179 41 41 41 62 62 62 F F F 9 180-199 38 38 38 57 57 57 F T T <--- 10 200-219 39 39 39 59 59 59 F T T <--- 11 220-239 29 29 29 46 46 46 F F F 12 240-259 32 32 32 49 49 49 F T T <--- 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 61 61 61 F F F 15 300-319 40 40 40 60 60 60 F T T <--- 16 320-339 38 38 38 58 58 58 F T T <--- 17 340-359 35 35 35 53 53 53 F T T <--- 18 360-379 40 40 40 60 60 60 F T T <--- 19 380-399 41 41 41 62 62 62 F T T <--- 20 400-419 39 39 39 59 59 59 F T T <--- 21 420-439 38 38 38 57 57 57 F F F 22 440-459 40 40 40 60 60 60 F F F 23 460-479 40 40 40 60 60 60 F F F 24 480-499 41 41 41 62 62 62 F F F 25 500-519 41 41 41 62 62 62 F F F 26 520-539 40 42 42 60 63 63 T F F <--- 27 540-559 40 42 42 60 63 63 F F F <--- 28 560-579 37 37 37 56 56 56 F T T <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 42, Max flush: 63 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 41 41 41 62 62 62 F F F 2 040-059 42 42 42 63 63 63 F F F 3 060-079 43 43 43 64 64 64 F F F 4 080-099 43 43 43 64 64 64 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 39 39 59 59 59 F F F 7 140-159 43 43 43 64 64 64 F F F 8 160-179 41 41 41 62 62 62 F F F 9 180-199 43 43 43 64 64 64 F F F 10 200-219 43 43 43 64 64 64 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 43 43 43 64 64 64 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 61 61 61 F F F 15 300-319 43 43 43 64 64 64 F F F 16 320-339 43 43 43 64 64 64 F F F 17 340-359 43 43 43 64 64 64 F F F 18 360-379 43 43 43 64 64 64 F F F 19 380-399 43 43 43 64 64 64 F F F 20 400-419 43 43 43 64 64 64 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 40 40 40 60 60 60 F F F 23 460-479 40 40 40 60 60 60 F F F 24 480-499 41 41 41 62 62 62 F F F 25 500-519 41 41 41 62 62 62 F F F 26 520-539 42 42 42 63 63 63 F F F 27 540-559 42 42 42 63 63 63 F F F 28 560-579 43 43 43 64 64 64 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 43, Max flush: 64 Begin loop 22 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.776642832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:22.795797811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.795897428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.819315029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.8199181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.819993969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.820020812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.820061422Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"Crucible bee8a3ef-c58e-45f4-9cf1-87af82d27927 has session id: 97b51018-8246-4a56-ac0b-394909b5fc22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.821389102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Upstairs opts: Upstairs UUID: bee8a3ef-c58e-45f4-9cf1-87af82d27927, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.821503939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Crucible stats registered with UUID: bee8a3ef-c58e-45f4-9cf1-87af82d27927","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.821540541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:22.821576202Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.821956081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822030468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822063363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822399516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822450557Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"bee8a3ef-c58e-45f4-9cf1-87af82d27927 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822585183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822618999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82264965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822683467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822719848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82275111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"ds_connection connected from Ok(127.0.0.1:62555)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822919703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82295884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"ds_connection connected from Ok(127.0.0.1:45637)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.822996965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"ds_connection connected from Ok(127.0.0.1:51317)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.823368418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82523418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.825439966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82646053Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82650112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.8265418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[0]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 59, 64, 62, 64, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827738602Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[0]R generation[0..12]: [39, 41, 42, 43, 43, 39, 39, 43, 41, 43, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827799813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827840864Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[1]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 59, 64, 62, 64, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827882446Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[1]R generation[0..12]: [39, 41, 42, 43, 43, 39, 39, 43, 41, 43, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827920491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.827960039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[2]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 59, 64, 62, 64, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828027813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[2]R generation[0..12]: [39, 41, 42, 43, 43, 39, 39, 43, 41, 43, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828068613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828113031Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828145445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828177558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Next flush: 65","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828209792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:01:22.828242457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"":"downstairs","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828286744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828331733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"0","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828366281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"1","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828398434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"client":"2","":"downstairs","session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828432141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828463192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"bee8a3ef-c58e-45f4-9cf1-87af82d27927 is now active with session: 97b51018-8246-4a56-ac0b-394909b5fc22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82849768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.82852781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368,"session_id":"97b51018-8246-4a56-ac0b-394909b5fc22"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828558801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} {"msg":"Activated sub_volume bee8a3ef-c58e-45f4-9cf1-87af82d27927","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:22.828586696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 494 len 24576 data: 4 4 5 6 6 7 02/30 Read block 487 len 4096 03/30 Write block 568 len 8192 data: 6 5 04/30 Flush 05/30 Flush 06/30 Flush 07/30 Write block 139 len 40960 data: 4 5 5 5 5 7 8 8 6 6 08/30 Write block 324 len 16384 data: 2 3 3 3 09/30 Read block 528 len 24576 10/30 Read block 541 len 16384 11/30 Read block 58 len 24576 12/30 Read block 142 len 32768 13/30 Read block 550 len 16384 14/30 Write block 241 len 32768 data: 3 4 4 4 3 2 2 3 15/30 Read block 228 len 16384 16/30 Read block 188 len 24576 17/30 Flush 18/30 Write block 240 len 20480 data: 4 4 5 5 5 19/30 Write block 366 len 12288 data: 4 4 2 20/30 Write block 295 len 32768 data: 5 4 4 4 4 4 5 5 21/30 Read block 34 len 12288 22/30 Write block 520 len 32768 data: 7 7 7 6 7 9 7 6 23/30 Write block 185 len 40960 data: 6 6 6 5 4 4 5 4 4 6 24/30 Read block 384 len 32768 25/30 Read block 583 len 8192 26/30 Read block 381 len 12288 27/30 Read block 78 len 16384 28/30 Read block 341 len 12288 29/30 Write block 378 len 36864 data: 8 9 9 8 6 5 6 5 5 30/30 Read block 217 len 8192 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 10 Done Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 0 0 23 Done 1 24 24 49 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:22.860959108Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1368} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 41 41 41 62 62 62 F F F 2 040-059 42 42 42 63 63 63 F F F 3 060-079 43 43 43 64 64 64 F F F 4 080-099 43 43 43 64 64 64 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 39 44 44 59 68 68 T F F <--- 7 140-159 43 44 44 64 68 68 T F F <--- 8 160-179 41 41 41 62 62 62 F F F 9 180-199 43 43 43 64 64 64 F T T <--- 10 200-219 43 43 43 64 64 64 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 43 44 44 64 68 68 F T T <--- 13 260-279 39 39 39 59 59 59 F F F 14 280-299 40 40 40 61 61 61 F T T <--- 15 300-319 43 43 43 64 64 64 F T T <--- 16 320-339 43 44 44 64 68 68 F F F <--- 17 340-359 43 43 43 64 64 64 F F F 18 360-379 43 43 43 64 64 64 F T T <--- 19 380-399 43 43 43 64 64 64 F T T <--- 20 400-419 43 43 43 64 64 64 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 40 40 40 60 60 60 F F F 23 460-479 40 40 40 60 60 60 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 41 41 41 62 62 62 F F F 26 520-539 42 42 42 63 63 63 F T T <--- 27 540-559 42 42 42 63 63 63 F F F 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 44, Max flush: 68 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 41 41 41 62 62 62 F F F 2 040-059 42 42 42 63 63 63 F F F 3 060-079 43 43 43 64 64 64 F F F 4 080-099 43 43 43 64 64 64 F F F 5 100-119 39 39 39 59 59 59 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 44 44 44 68 68 68 F F F 8 160-179 41 41 41 62 62 62 F F F 9 180-199 45 45 45 69 69 69 F F F 10 200-219 43 43 43 64 64 64 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 39 39 39 59 59 59 F F F 14 280-299 45 45 45 69 69 69 F F F 15 300-319 45 45 45 69 69 69 F F F 16 320-339 44 44 44 68 68 68 F F F 17 340-359 43 43 43 64 64 64 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 45 45 45 69 69 69 F F F 20 400-419 43 43 43 64 64 64 F F F 21 420-439 38 38 38 57 57 57 F F F 22 440-459 40 40 40 60 60 60 F F F 23 460-479 40 40 40 60 60 60 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 41 41 41 62 62 62 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 42 42 42 63 63 63 F F F 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 45, Max flush: 69 Begin loop 23 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.482044005Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.504655947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.505338574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.505423241Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.505445946Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.505477809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"Crucible 580740f0-aa56-4e2d-b509-57c6f34388d7 has session id: c2451f26-2700-45b6-82ca-d4d4878f59b0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.506865477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Upstairs opts: Upstairs UUID: 580740f0-aa56-4e2d-b509-57c6f34388d7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.506949683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Crucible stats registered with UUID: 580740f0-aa56-4e2d-b509-57c6f34388d7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50698865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:27.507012166Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.507421534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.507477625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.507598443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.508406929Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50845824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"580740f0-aa56-4e2d-b509-57c6f34388d7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.5086Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50864062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50867632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50871151Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.508746629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.508775546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.509031812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:52998)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.509150206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:55433)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.50919829Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:50528)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.509254311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.51119475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.511453862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.511910262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.512169574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.512666104Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[0]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 68, 68, 62, 69, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513245637Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[0]R generation[0..12]: [39, 41, 42, 43, 43, 39, 44, 44, 41, 45, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.51330288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513351747Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[1]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 68, 68, 62, 69, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513396986Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[1]R generation[0..12]: [39, 41, 42, 43, 43, 39, 44, 44, 41, 45, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513440452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513483346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[2]R flush_numbers[0..12]: [59, 62, 63, 64, 64, 59, 68, 68, 62, 69, 64, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513524688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[2]R generation[0..12]: [39, 41, 42, 43, 43, 39, 44, 44, 41, 45, 43, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513566791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513628884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513669043Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513721436Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Next flush: 70","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513757788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513793048Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"":"downstairs","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513830872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513867344Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"0","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513908315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"1","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513950218Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"client":"2","":"downstairs","session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.513992902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.514031729Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"580740f0-aa56-4e2d-b509-57c6f34388d7 is now active with session: c2451f26-2700-45b6-82ca-d4d4878f59b0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.514086958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.51412346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376,"session_id":"c2451f26-2700-45b6-82ca-d4d4878f59b0"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.514160563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} {"msg":"Activated sub_volume 580740f0-aa56-4e2d-b509-57c6f34388d7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:27.514193798Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 180 len 4096 data: 4 02/30 Read block 118 len 28672 03/30 Read block 465 len 28672 04/30 Read block 69 len 36864 05/30 Write block 266 len 28672 data: 4 4 3 3 3 4 6 06/30 Read block 503 len 16384 07/30 Read block 503 len 36864 08/30 Write block 150 len 28672 data: 5 6 4 4 4 3 4 09/30 Read block 98 len 32768 10/30 Read block 480 len 32768 11/30 Write block 95 len 8192 data: 2 2 12/30 Flush 13/30 Write block 171 len 32768 data: 7 5 4 5 5 4 3 3 14/30 Write block 502 len 28672 data: 6 5 6 4 4 5 4 15/30 Flush 16/30 Write block 466 len 4096 data: 4 17/30 Read block 114 len 8192 18/30 Read block 463 len 24576 19/30 Write block 32 len 36864 data: 4 3 4 4 3 4 4 4 4 20/30 Flush 21/30 Write block 508 len 32768 data: 5 6 5 5 3 4 4 4 22/30 Write block 105 len 4096 data: 3 23/30 Write block 337 len 20480 data: 3 3 3 3 3 24/30 Read block 270 len 40960 25/30 Write block 201 len 28672 data: 3 5 5 5 5 6 5 26/30 Write block 274 len 36864 data: 4 5 6 7 7 7 9 9 9 27/30 Write block 438 len 24576 data: 5 4 4 6 6 6 28/30 Write block 72 len 16384 data: 5 7 8 9 29/30 Read block 363 len 36864 30/30 Write block 386 len 36864 data: 6 5 5 4 5 3 3 4 3 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 7 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:27.54829364Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1376} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 46 41 46 72 62 72 F F F <--- 2 040-059 46 42 46 72 63 72 F F F <--- 3 060-079 43 43 43 64 64 64 T F T <--- 4 080-099 46 43 46 70 64 70 F F F <--- 5 100-119 39 39 39 59 59 59 T F T <--- 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 44 46 70 68 70 F F F <--- 8 160-179 46 41 46 71 62 71 F F F <--- 9 180-199 46 45 46 70 69 70 F T F <--- 10 200-219 43 43 43 64 64 64 T F T <--- 11 220-239 29 29 29 46 46 46 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 46 39 46 70 59 70 T F T <--- 14 280-299 45 45 45 69 69 69 T F T <--- 15 300-319 45 45 45 69 69 69 F F F 16 320-339 44 44 44 68 68 68 T F T <--- 17 340-359 43 43 43 64 64 64 T F T <--- 18 360-379 45 45 45 69 69 69 F F F 19 380-399 45 45 45 69 69 69 T F T <--- 20 400-419 43 43 43 64 64 64 F F F 21 420-439 38 38 38 57 57 57 T F T <--- 22 440-459 40 40 40 60 60 60 T F T <--- 23 460-479 46 40 46 72 60 72 F F F <--- 24 480-499 44 44 44 65 65 65 F F F 25 500-519 46 41 46 71 62 71 T F T <--- 26 520-539 45 45 45 69 69 69 F F F 27 540-559 42 42 42 63 63 63 F F F 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 46, Max flush: 72 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F F F 1 020-039 46 46 46 72 72 72 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 47 47 47 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F F F 5 100-119 47 47 47 73 73 73 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 47 47 47 73 73 73 F F F 11 220-239 29 29 29 46 46 46 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 47 47 47 73 73 73 F F F 14 280-299 47 47 47 73 73 73 F F F 15 300-319 45 45 45 69 69 69 F F F 16 320-339 47 47 47 73 73 73 F F F 17 340-359 47 47 47 73 73 73 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 47 47 47 73 73 73 F F F 20 400-419 43 43 43 64 64 64 F F F 21 420-439 47 47 47 73 73 73 F F F 22 440-459 47 47 47 73 73 73 F F F 23 460-479 46 46 46 72 72 72 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 47 47 47 73 73 73 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 42 42 42 63 63 63 F F F 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 47, Max flush: 73 Begin loop 24 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.217367034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:32.237167156Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.237239028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.255360689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.255955452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.256099266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.256135017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.25618223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"Crucible 29855e39-24dd-4589-8b87-509c1cb78efe has session id: b917c11e-236c-4aee-98bb-b63add1bd32a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.257103278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Upstairs opts: Upstairs UUID: 29855e39-24dd-4589-8b87-509c1cb78efe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.25723112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Crucible stats registered with UUID: 29855e39-24dd-4589-8b87-509c1cb78efe","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.257266931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:32.257293473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.257827687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.257916412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.257943636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258195834Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258240101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"29855e39-24dd-4589-8b87-509c1cb78efe active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258441589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258470365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258495415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258520093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258544582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258569771Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.2585946Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:42817)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.258891567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:39976)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.259234263Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"ds_connection connected from Ok(127.0.0.1:59386)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.259325854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.260921132Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.261118942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.261151857Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262061863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262196388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[0]R flush_numbers[0..12]: [59, 72, 72, 73, 70, 73, 68, 70, 71, 70, 73, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262345562Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[0]R generation[0..12]: [39, 46, 46, 47, 46, 47, 44, 46, 46, 46, 47, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262377496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262403176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[1]R flush_numbers[0..12]: [59, 72, 72, 73, 70, 73, 68, 70, 71, 70, 73, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262426292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[1]R generation[0..12]: [39, 46, 46, 47, 46, 47, 44, 46, 46, 46, 47, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262449377Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.26247585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[2]R flush_numbers[0..12]: [59, 72, 72, 73, 70, 73, 68, 70, 71, 70, 73, 46]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262499005Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[2]R generation[0..12]: [39, 46, 46, 47, 46, 47, 44, 46, 46, 46, 47, 29]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262522291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262546519Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262566809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262587811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Next flush: 74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.26260794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.26262813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"":"downstairs","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262648731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262668089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"0","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262694301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"1","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262716044Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"client":"2","":"downstairs","session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262737616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Set Downstairs and Upstairs active after reconciliation"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262769379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"29855e39-24dd-4589-8b87-509c1cb78efe is now active with session: b917c11e-236c-4aee-98bb-b63add1bd32a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262793206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262812203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384,"session_id":"b917c11e-236c-4aee-98bb-b63add1bd32a"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262832303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} {"msg":"Activated sub_volume 29855e39-24dd-4589-8b87-509c1cb78efe","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:32.262850158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 288 len 20480 data: 5 7 5 5 5 02/30 Read block 0 len 12288 03/30 Write block 273 len 16384 data: 6 5 6 7 04/30 Read block 42 len 32768 05/30 Flush 06/30 Write block 260 len 40960 data: 7 7 7 6 6 4 5 5 4 4 07/30 Read block 568 len 4096 08/30 Read block 215 len 40960 09/30 Read block 319 len 32768 10/30 Read block 391 len 20480 11/30 Write block 10 len 16384 data: 5 8 8 8 12/30 Read block 423 len 8192 13/30 Read block 63 len 20480 14/30 Read block 107 len 12288 15/30 Write block 457 len 4096 data: 2 16/30 Write block 289 len 24576 data: 8 6 6 6 4 4 17/30 Write block 82 len 36864 data: 11 9 8 10 8 7 8 8 6 18/30 Read block 48 len 20480 19/30 Write block 17 len 4096 data: 4 20/30 Read block 115 len 40960 21/30 Read block 311 len 16384 22/30 Read block 590 len 28672 23/30 Read block 375 len 16384 24/30 Read block 342 len 16384 25/30 Write block 545 len 36864 data: 4 3 5 5 7 7 6 6 4 26/30 Write block 234 len 8192 data: 4 4 27/30 Read block 3 len 32768 28/30 Write block 92 len 16384 data: 5 4 2 3 29/30 Write block 88 len 32768 data: 9 9 7 5 6 5 3 4 30/30 Read block 197 len 4096 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Read 3 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Write 8 Sent Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 2 32 Done 0 30 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:32.293376027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1384} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 59 59 59 F T T <--- 1 020-039 46 46 46 72 72 72 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 47 47 47 73 73 73 F F F 4 080-099 46 46 46 70 70 70 F T T <--- 5 100-119 47 47 47 73 73 73 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 47 47 47 73 73 73 F F F 11 220-239 29 29 29 46 46 46 F T T <--- 12 240-259 45 45 45 69 69 69 F F F 13 260-279 47 48 48 73 74 74 F T T <--- 14 280-299 47 48 48 73 74 74 F T T <--- 15 300-319 45 45 45 69 69 69 F F F 16 320-339 47 47 47 73 73 73 F F F 17 340-359 47 47 47 73 73 73 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 47 47 47 73 73 73 F F F 20 400-419 43 43 43 64 64 64 F F F 21 420-439 47 47 47 73 73 73 F F F 22 440-459 47 47 47 73 73 73 F T T <--- 23 460-479 46 46 46 72 72 72 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 47 47 47 73 73 73 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 42 42 42 63 63 63 F T T <--- 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 48, Max flush: 74 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 46 46 46 72 72 72 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 47 47 47 73 73 73 F F F 4 080-099 49 49 49 75 75 75 F F F 5 100-119 47 47 47 73 73 73 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 47 47 47 73 73 73 F F F 11 220-239 49 49 49 75 75 75 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 49 49 49 75 75 75 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 45 45 45 69 69 69 F F F 16 320-339 47 47 47 73 73 73 F F F 17 340-359 47 47 47 73 73 73 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 47 47 47 73 73 73 F F F 20 400-419 43 43 43 64 64 64 F F F 21 420-439 47 47 47 73 73 73 F F F 22 440-459 49 49 49 75 75 75 F F F 23 460-479 46 46 46 72 72 72 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 47 47 47 73 73 73 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 49 49 49 75 75 75 F F F 28 560-579 44 44 44 65 65 65 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 49, Max flush: 75 Begin loop 25 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.853734607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.876772449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.877266253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.877349307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.877374507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.87741162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"Crucible b2cbede3-ce51-42db-a240-d42152ddb9f2 has session id: 902bfbdf-7e72-4305-b0be-c1f5507eed16","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.878520669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Upstairs opts: Upstairs UUID: b2cbede3-ce51-42db-a240-d42152ddb9f2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.878606237Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Crucible stats registered with UUID: b2cbede3-ce51-42db-a240-d42152ddb9f2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.878655475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:36.878682809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879055995Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879120442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879150742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879391828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879427438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"b2cbede3-ce51-42db-a240-d42152ddb9f2 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879542395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879581382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879620238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879662512Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.879705497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.880105526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.880188109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"ds_connection connected from Ok(127.0.0.1:39312)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.880216355Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"ds_connection connected from Ok(127.0.0.1:64043)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.880242827Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"ds_connection connected from Ok(127.0.0.1:55671)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.880446439Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.88171379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.882102988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.882259396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.882522385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.882816265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[0]R flush_numbers[0..12]: [75, 72, 72, 73, 75, 73, 68, 70, 71, 70, 73, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883442932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[0]R generation[0..12]: [49, 46, 46, 47, 49, 47, 44, 46, 46, 46, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883483963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883513742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[1]R flush_numbers[0..12]: [75, 72, 72, 73, 75, 73, 68, 70, 71, 70, 73, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883541136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[1]R generation[0..12]: [49, 46, 46, 47, 49, 47, 44, 46, 46, 46, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.8836192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883662315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[2]R flush_numbers[0..12]: [75, 72, 72, 73, 75, 73, 68, 70, 71, 70, 73, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883689489Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[2]R generation[0..12]: [49, 46, 46, 47, 49, 47, 44, 46, 46, 46, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883716382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-06T06:01:36.883763324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883789636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883814345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Next flush: 76","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.88383709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883860215Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"":"downstairs","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883884443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883906607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"0","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883931526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"1","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883955524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"client":"2","":"downstairs","session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.883979962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.884002907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"b2cbede3-ce51-42db-a240-d42152ddb9f2 is now active with session: 902bfbdf-7e72-4305-b0be-c1f5507eed16","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.884029019Z","hostname":"bmat-EVT22200007-0000ec3a","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.884061803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392,"session_id":"902bfbdf-7e72-4305-b0be-c1f5507eed16"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.884085149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} {"msg":"Activated sub_volume b2cbede3-ce51-42db-a240-d42152ddb9f2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:36.884105499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 45 len 32768 data: 5 5 4 5 4 5 4 3 03/30 Flush 04/30 Write block 302 len 12288 data: 6 5 2 05/30 Write block 381 len 16384 data: 9 7 6 7 06/30 Write block 206 len 4096 data: 7 07/30 Read block 587 len 20480 08/30 Read block 507 len 24576 09/30 Write block 501 len 32768 data: 7 7 6 7 5 5 6 6 10/30 Read block 403 len 24576 11/30 Read block 78 len 24576 12/30 Read block 385 len 24576 13/30 Read block 167 len 20480 14/30 Read block 204 len 28672 15/30 Write block 414 len 36864 data: 6 5 5 5 4 4 5 5 5 16/30 Write block 420 len 20480 data: 6 6 6 6 6 17/30 Flush 18/30 Write block 567 len 16384 data: 6 7 6 7 19/30 Write block 66 len 28672 data: 8 7 9 6 5 4 6 20/30 Write block 465 len 4096 data: 5 21/30 Read block 220 len 12288 22/30 Write block 112 len 16384 data: 4 6 4 6 23/30 Write block 39 len 4096 data: 5 24/30 Write block 400 len 28672 data: 6 7 8 10 7 8 6 25/30 Write block 457 len 24576 data: 3 3 3 4 4 4 26/30 Read block 466 len 28672 27/30 Write block 458 len 28672 data: 4 4 5 5 5 5 4 28/30 Write block 306 len 16384 data: 2 2 3 3 29/30 Write block 468 len 20480 data: 5 4 4 4 3 30/30 Read block 259 len 28672 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 29 1 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1000)) Some(JobId(1016)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:36.9124989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1392} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 46 46 46 72 72 72 T F T <--- 2 040-059 50 46 50 77 72 77 F T F <--- 3 060-079 47 47 47 73 73 73 T F T <--- 4 080-099 49 49 49 75 75 75 F F F 5 100-119 47 47 47 73 73 73 T F T <--- 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 50 47 50 78 73 78 F F F <--- 11 220-239 49 49 49 75 75 75 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 49 49 49 75 75 75 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 50 45 50 78 69 78 T F T <--- 16 320-339 47 47 47 73 73 73 F F F 17 340-359 47 47 47 73 73 73 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 50 47 50 78 73 78 F F F <--- 20 400-419 50 43 50 78 64 78 T F T <--- 21 420-439 50 47 50 78 73 78 F F F <--- 22 440-459 49 49 49 75 75 75 T F T <--- 23 460-479 46 46 46 72 72 72 T F T <--- 24 480-499 44 44 44 65 65 65 F F F 25 500-519 50 47 50 78 73 78 F F F <--- 26 520-539 45 45 45 69 69 69 F F F 27 540-559 49 49 49 75 75 75 F F F 28 560-579 44 44 44 65 65 65 T F T <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 50, Max flush: 78 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 51 51 51 79 79 79 F F F 2 040-059 50 50 50 77 77 77 F F F 3 060-079 51 51 51 79 79 79 F F F 4 080-099 49 49 49 75 75 75 F F F 5 100-119 51 51 51 79 79 79 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 50 50 50 78 78 78 F F F 11 220-239 49 49 49 75 75 75 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 49 49 49 75 75 75 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 51 51 51 79 79 79 F F F 16 320-339 47 47 47 73 73 73 F F F 17 340-359 47 47 47 73 73 73 F F F 18 360-379 45 45 45 69 69 69 F F F 19 380-399 50 50 50 78 78 78 F F F 20 400-419 51 51 51 79 79 79 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 51 51 51 79 79 79 F F F 23 460-479 51 51 51 79 79 79 F F F 24 480-499 44 44 44 65 65 65 F F F 25 500-519 50 50 50 78 78 78 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 49 49 49 75 75 75 F F F 28 560-579 51 51 51 79 79 79 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 51, Max flush: 79 Begin loop 26 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.498786037Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.52605063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.526816882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.526902932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.526929103Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.526967259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"Crucible 2081bada-2f45-4cb2-ac97-71a72808cb94 has session id: 6d491aad-6a9e-44b0-a801-297ea6ba6ced","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.528292043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Upstairs opts: Upstairs UUID: 2081bada-2f45-4cb2-ac97-71a72808cb94, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.528358484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Crucible stats registered with UUID: 2081bada-2f45-4cb2-ac97-71a72808cb94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.528389375Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:41.528416158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.52889363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.528986223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529020821Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529136079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529167641Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"2081bada-2f45-4cb2-ac97-71a72808cb94 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529296235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.52932939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529367525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529398627Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529429047Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529460709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.5295321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"ds_connection connected from Ok(127.0.0.1:41509)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.529816872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"ds_connection connected from Ok(127.0.0.1:42652)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.530111544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"ds_connection connected from Ok(127.0.0.1:58492)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.530197203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.531964781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532052965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532186338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532631286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532698679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[0]R flush_numbers[0..12]: [75, 79, 77, 79, 75, 79, 68, 70, 71, 70, 78, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.53287638Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[0]R generation[0..12]: [49, 51, 50, 51, 49, 51, 44, 46, 46, 46, 50, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.53291718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532962109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[1]R flush_numbers[0..12]: [75, 79, 77, 79, 75, 79, 68, 70, 71, 70, 78, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.532996637Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[1]R generation[0..12]: [49, 51, 50, 51, 49, 51, 44, 46, 46, 46, 50, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533031315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533067677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[2]R flush_numbers[0..12]: [75, 79, 77, 79, 75, 79, 68, 70, 71, 70, 78, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533101784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[2]R generation[0..12]: [49, 51, 50, 51, 49, 51, 44, 46, 46, 46, 50, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533135451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533171602Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2026-03-06T06:01:41.533201571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533246811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Next flush: 80","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533276018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533305206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"":"downstairs","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533336107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533364122Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"0","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533430744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"1","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533471023Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"client":"2","":"downstairs","session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533506964Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533536322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"2081bada-2f45-4cb2-ac97-71a72808cb94 is now active with session: 6d491aad-6a9e-44b0-a801-297ea6ba6ced","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533569337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533597002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400,"session_id":"6d491aad-6a9e-44b0-a801-297ea6ba6ced"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.533626219Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} {"msg":"Activated sub_volume 2081bada-2f45-4cb2-ac97-71a72808cb94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:41.53365166Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 213 len 4096 data: 3 02/30 Write block 353 len 4096 data: 5 03/30 Flush 04/30 Write block 265 len 16384 data: 5 6 6 5 05/30 Flush 06/30 Write block 82 len 28672 data: 12 10 9 11 9 8 10 07/30 Write block 261 len 8192 data: 8 8 08/30 Flush 09/30 Write block 226 len 24576 data: 4 4 4 3 4 4 10/30 Read block 507 len 32768 11/30 Read block 390 len 24576 12/30 Read block 387 len 16384 13/30 Read block 333 len 28672 14/30 Read block 409 len 32768 15/30 Read block 463 len 32768 16/30 Read block 261 len 40960 17/30 Read block 199 len 36864 18/30 Flush 19/30 Write block 37 len 4096 data: 5 20/30 Write block 483 len 28672 data: 6 6 5 4 5 4 5 21/30 Write block 200 len 4096 data: 4 22/30 Flush 23/30 Flush 24/30 Write block 305 len 24576 data: 2 3 3 4 4 3 25/30 Read block 429 len 32768 26/30 Write block 364 len 4096 data: 5 27/30 Read block 310 len 32768 28/30 Write block 458 len 12288 data: 5 5 6 29/30 Read block 355 len 28672 30/30 Read block 384 len 40960 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Sent Done false 1001 Acked Write 1 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:41.563410495Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1400} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 52 51 52 84 79 84 F F F <--- 2 040-059 50 50 50 77 77 77 F F F 3 060-079 51 51 51 79 79 79 F F F 4 080-099 52 49 52 82 75 82 F F F <--- 5 100-119 51 51 51 79 79 79 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 50 52 84 78 84 F F F <--- 11 220-239 52 49 52 83 75 83 F F F <--- 12 240-259 45 45 45 69 69 69 F F F 13 260-279 52 49 52 82 75 82 F F F <--- 14 280-299 49 49 49 75 75 75 F F F 15 300-319 51 51 51 79 79 79 T F T <--- 16 320-339 47 47 47 73 73 73 F F F 17 340-359 52 47 52 80 73 80 F F F <--- 18 360-379 45 45 45 69 69 69 T F T <--- 19 380-399 50 50 50 78 78 78 F F F 20 400-419 51 51 51 79 79 79 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 51 51 51 79 79 79 T F T <--- 23 460-479 51 51 51 79 79 79 T F T <--- 24 480-499 52 44 52 84 65 84 F F F <--- 25 500-519 50 50 50 78 78 78 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 49 49 49 75 75 75 F F F 28 560-579 51 51 51 79 79 79 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 52, Max flush: 84 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 52 52 52 84 84 84 F F F 2 040-059 50 50 50 77 77 77 F F F 3 060-079 51 51 51 79 79 79 F F F 4 080-099 52 52 52 82 82 82 F F F 5 100-119 51 51 51 79 79 79 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 52 52 84 84 84 F F F 11 220-239 52 52 52 83 83 83 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 53 53 53 85 85 85 F F F 16 320-339 47 47 47 73 73 73 F F F 17 340-359 52 52 52 80 80 80 F F F 18 360-379 53 53 53 85 85 85 F F F 19 380-399 50 50 50 78 78 78 F F F 20 400-419 51 51 51 79 79 79 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 53 53 53 85 85 85 F F F 23 460-479 53 53 53 85 85 85 F F F 24 480-499 52 52 52 84 84 84 F F F 25 500-519 50 50 50 78 78 78 F F F 26 520-539 45 45 45 69 69 69 F F F 27 540-559 49 49 49 75 75 75 F F F 28 560-579 51 51 51 79 79 79 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 53, Max flush: 85 Begin loop 27 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.117708455Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:46.139443346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.139524276Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.159016501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.159547579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.159627306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.159656003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.159696854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"Crucible e33f6df3-d1e3-47dd-9897-7300ca4eef45 has session id: 19c5dd3b-b51b-4d43-84b7-e46546189e0e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.160631388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Upstairs opts: Upstairs UUID: e33f6df3-d1e3-47dd-9897-7300ca4eef45, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.160692158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Crucible stats registered with UUID: e33f6df3-d1e3-47dd-9897-7300ca4eef45","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.160728419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:46.160755092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16110405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161158288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161187726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16142291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161453871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"e33f6df3-d1e3-47dd-9897-7300ca4eef45 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161568677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161599709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161627974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161659467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16169139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161821366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61596)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161857217Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.161885653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55212)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.162167489Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54602)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.162626475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.164046026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.164107978Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.164386137Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165399607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165466399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[0]R flush_numbers[0..12]: [75, 84, 77, 79, 82, 79, 68, 70, 71, 70, 84, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165501899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[0]R generation[0..12]: [49, 52, 50, 51, 52, 51, 44, 46, 46, 46, 52, 52]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165534854Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165568601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[1]R flush_numbers[0..12]: [75, 84, 77, 79, 82, 79, 68, 70, 71, 70, 84, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165599943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[1]R generation[0..12]: [49, 52, 50, 51, 52, 51, 44, 46, 46, 46, 52, 52]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165630994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165664059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[2]R flush_numbers[0..12]: [75, 84, 77, 79, 82, 79, 68, 70, 71, 70, 84, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16569502Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[2]R generation[0..12]: [49, 52, 50, 51, 52, 51, 44, 46, 46, 46, 52, 52]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165725801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165758666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165792733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165821019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Next flush: 86","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165847862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165874785Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"":"downstairs","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16590279Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165928431Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"0","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165956787Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"1","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.165984692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"client":"2","":"downstairs","session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.16603942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.166066784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"e33f6df3-d1e3-47dd-9897-7300ca4eef45 is now active with session: 19c5dd3b-b51b-4d43-84b7-e46546189e0e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.166097134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-06T06:01:46.166122804Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408,"session_id":"19c5dd3b-b51b-4d43-84b7-e46546189e0e"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.166161901Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} {"msg":"Activated sub_volume e33f6df3-d1e3-47dd-9897-7300ca4eef45","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:46.166184987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 225 len 32768 data: 3 5 5 5 4 5 5 4 02/30 Read block 45 len 32768 03/30 Flush 04/30 Write block 333 len 28672 data: 4 5 5 5 4 4 4 05/30 Read block 487 len 12288 06/30 Read block 293 len 4096 07/30 Flush 08/30 Write block 302 len 24576 data: 7 6 3 3 4 4 09/30 Write block 313 len 32768 data: 6 6 7 5 4 4 4 4 10/30 Write block 552 len 16384 data: 7 5 5 5 11/30 Read block 412 len 32768 12/30 Write block 29 len 40960 data: 6 5 4 5 4 5 5 4 6 5 13/30 Read block 225 len 20480 14/30 Read block 305 len 28672 15/30 Write block 567 len 20480 data: 7 8 7 8 5 16/30 Write block 399 len 28672 data: 6 7 8 9 11 8 9 17/30 Write block 53 len 36864 data: 4 6 7 6 6 5 5 6 7 18/30 Write block 445 len 28672 data: 7 7 5 6 5 5 4 19/30 Read block 423 len 20480 20/30 Read block 213 len 20480 21/30 Read block 472 len 28672 22/30 Read block 59 len 32768 23/30 Read block 126 len 4096 24/30 Write block 441 len 16384 data: 7 7 7 6 25/30 Read block 8 len 24576 26/30 Write block 356 len 4096 data: 4 27/30 Write block 519 len 28672 data: 7 8 8 8 7 8 10 28/30 Read block 31 len 28672 29/30 Write block 228 len 40960 data: 6 5 6 6 5 3 5 5 4 5 30/30 Read block 198 len 4096 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 3 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 10 Sent Sent Done false 1029 Acked Read 1 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 0 30 Done 2 28 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:46.200239489Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1408} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 52 52 52 84 84 84 F T T <--- 2 040-059 50 50 50 77 77 77 F T T <--- 3 060-079 51 51 51 79 79 79 F T T <--- 4 080-099 52 52 52 82 82 82 F F F 5 100-119 51 51 51 79 79 79 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 52 52 84 84 84 F F F 11 220-239 52 54 54 83 86 86 T T T <--- 12 240-259 45 45 45 69 69 69 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 53 53 53 85 85 85 F T T <--- 16 320-339 47 54 54 73 87 87 F T T <--- 17 340-359 52 52 52 80 80 80 F T T <--- 18 360-379 53 53 53 85 85 85 F F F 19 380-399 50 50 50 78 78 78 F T T <--- 20 400-419 51 51 51 79 79 79 F T T <--- 21 420-439 50 50 50 78 78 78 F F F 22 440-459 53 53 53 85 85 85 F T T <--- 23 460-479 53 53 53 85 85 85 F F F 24 480-499 52 52 52 84 84 84 F F F 25 500-519 50 50 50 78 78 78 F T T <--- 26 520-539 45 45 45 69 69 69 F T T <--- 27 540-559 49 49 49 75 75 75 F T T <--- 28 560-579 51 51 51 79 79 79 F T T <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 54, Max flush: 87 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 75 75 75 F F F 1 020-039 55 55 55 88 88 88 F F F 2 040-059 55 55 55 88 88 88 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 52 52 52 82 82 82 F F F 5 100-119 51 51 51 79 79 79 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 46 46 46 71 71 71 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 52 52 84 84 84 F F F 11 220-239 55 55 55 88 88 88 F F F 12 240-259 45 45 45 69 69 69 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 49 49 49 75 75 75 F F F 15 300-319 55 55 55 88 88 88 F F F 16 320-339 55 55 55 88 88 88 F F F 17 340-359 55 55 55 88 88 88 F F F 18 360-379 53 53 53 85 85 85 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 55 55 88 88 88 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 55 55 55 88 88 88 F F F 23 460-479 53 53 53 85 85 85 F F F 24 480-499 52 52 52 84 84 84 F F F 25 500-519 55 55 55 88 88 88 F F F 26 520-539 55 55 55 88 88 88 F F F 27 540-559 55 55 55 88 88 88 F F F 28 560-579 55 55 55 88 88 88 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 55, Max flush: 88 Begin loop 28 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.812427548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.835653611Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.836415074Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.836503288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.836528147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.83656516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"Crucible e8f50d39-dc74-4866-8d3d-db1f070b2de1 has session id: fb773f4e-19da-40c2-a78f-dd5be7f3c42c","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.8377055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Upstairs opts: Upstairs UUID: e8f50d39-dc74-4866-8d3d-db1f070b2de1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.837769387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Crucible stats registered with UUID: e8f50d39-dc74-4866-8d3d-db1f070b2de1","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.837800728Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:50.837827281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.838282199Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.838616699Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.83865296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.838918985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.838957441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"e8f50d39-dc74-4866-8d3d-db1f070b2de1 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839072007Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839120473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839150101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839172405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839193547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839319375Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839347942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"ds_connection connected from Ok(127.0.0.1:32904)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839451456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55624)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839479902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45059)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.839519029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.840889464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.841108376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.841820531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.841988011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842024523Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[0]R flush_numbers[0..12]: [75, 88, 88, 88, 82, 79, 68, 70, 71, 70, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842835985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[0]R generation[0..12]: [49, 55, 55, 55, 52, 51, 44, 46, 46, 46, 52, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842885182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842925962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[1]R flush_numbers[0..12]: [75, 88, 88, 88, 82, 79, 68, 70, 71, 70, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842957865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[1]R generation[0..12]: [49, 55, 55, 55, 52, 51, 44, 46, 46, 46, 52, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.842988736Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843055849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[2]R flush_numbers[0..12]: [75, 88, 88, 88, 82, 79, 68, 70, 71, 70, 84, 88]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.84309197Z","Wait for a query_work_queue command to finish before sending IO hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[2]R generation[0..12]: [49, 55, 55, 55, 52, 51, 44, 46, 46, 46, 52, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843133652Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843167148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843194402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843222648Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Next flush: 89","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843248829Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843275552Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"":"downstairs","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843303567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843329639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"0","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843357404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"1","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843385299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"client":"2","":"downstairs","session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843417061Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843443644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"e8f50d39-dc74-4866-8d3d-db1f070b2de1 is now active with session: fb773f4e-19da-40c2-a78f-dd5be7f3c42c","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-06T06:01:50.843473443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843509213Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416,"session_id":"fb773f4e-19da-40c2-a78f-dd5be7f3c42c"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843536177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} {"msg":"Activated sub_volume e8f50d39-dc74-4866-8d3d-db1f070b2de1","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:50.843559142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 325 len 24576 data: 4 4 4 4 4 4 02/30 Flush 03/30 Flush 04/30 Write block 28 len 20480 data: 5 7 6 5 6 05/30 Write block 295 len 36864 data: 6 5 5 5 5 5 6 8 7 06/30 Read block 394 len 32768 07/30 Read block 129 len 24576 08/30 Write block 166 len 4096 data: 7 09/30 Write block 356 len 8192 data: 5 4 10/30 Read block 518 len 40960 11/30 Read block 499 len 16384 12/30 Read block 132 len 12288 13/30 Write block 327 len 16384 data: 5 5 5 5 14/30 Write block 355 len 4096 data: 4 15/30 Write block 333 len 4096 data: 5 16/30 Read block 429 len 24576 17/30 Flush 18/30 Write block 569 len 16384 data: 8 9 6 5 19/30 Write block 99 len 32768 data: 4 4 4 4 4 4 4 3 20/30 Write block 484 len 8192 data: 7 6 21/30 Write block 8 len 32768 data: 3 4 6 9 9 9 6 6 22/30 Read block 267 len 36864 23/30 Flush 24/30 Write block 449 len 4096 data: 6 25/30 Write block 236 len 20480 data: 5 6 5 4 5 26/30 Read block 307 len 4096 27/30 Write block 461 len 20480 data: 6 6 6 5 6 28/30 Write block 33 len 32768 data: 5 6 6 5 7 6 6 5 29/30 Write block 453 len 24576 data: 3 2 3 2 4 6 30/30 Write block 525 len 32768 data: 11 8 7 6 5 5 6 8 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 5 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 30 36 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:50.869107558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1416} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 49 92 92 75 F F F <--- 1 020-039 56 56 55 91 91 88 T T F <--- 2 040-059 55 55 55 88 88 88 T T F <--- 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 52 92 92 82 F F F <--- 5 100-119 56 56 51 92 92 79 F F F <--- 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 56 56 46 91 91 71 F F F <--- 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 52 52 84 84 84 F F F 11 220-239 55 55 55 88 88 88 T T F <--- 12 240-259 45 45 45 69 69 69 T T F <--- 13 260-279 52 52 52 82 82 82 F F F 14 280-299 56 56 49 91 91 75 F F F <--- 15 300-319 56 56 55 91 91 88 F F F <--- 16 320-339 56 56 55 91 91 88 F F F <--- 17 340-359 56 56 55 91 91 88 F F F <--- 18 360-379 53 53 53 85 85 85 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 55 55 88 88 88 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 55 55 55 88 88 88 T T F <--- 23 460-479 53 53 53 85 85 85 T T F <--- 24 480-499 56 56 52 92 92 84 F F F <--- 25 500-519 55 55 55 88 88 88 F F F 26 520-539 55 55 55 88 88 88 T T F <--- 27 540-559 55 55 55 88 88 88 F F F 28 560-579 56 56 55 92 92 88 F F F <--- 29 580-599 31 31 31 48 48 48 F F F Max gen: 56, Max flush: 92 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 56 56 92 92 92 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 56 56 56 92 92 92 F F F 6 120-139 44 44 44 68 68 68 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 56 56 56 91 91 91 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 52 52 84 84 84 F F F 11 220-239 57 57 57 93 93 93 F F F 12 240-259 57 57 57 93 93 93 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 56 56 56 91 91 91 F F F 15 300-319 56 56 56 91 91 91 F F F 16 320-339 56 56 56 91 91 91 F F F 17 340-359 56 56 56 91 91 91 F F F 18 360-379 53 53 53 85 85 85 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 55 55 88 88 88 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 57 57 57 93 93 93 F F F 23 460-479 57 57 57 93 93 93 F F F 24 480-499 56 56 56 92 92 92 F F F 25 500-519 55 55 55 88 88 88 F F F 26 520-539 57 57 57 93 93 93 F F F 27 540-559 55 55 55 88 88 88 F F F 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 57, Max flush: 93 Begin loop 29 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.507468727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:55.525983353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.526062188Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.54228392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.542911428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.542986506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.543012798Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.543050533Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"Crucible af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d has session id: 8d246e64-21b8-4693-9b86-942ed69d209d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544187156Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Upstairs opts: Upstairs UUID: af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544255991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Crucible stats registered with UUID: af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544296301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:55.544323555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544741249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544806558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.544839503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.545063786Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.545097933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.545229563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.545260003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.545288118Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546242351Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546291017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546314884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63319)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.54633805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:58942)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546360845Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546382227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63887)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.546412347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.547977966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.548274281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.548769328Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.548799868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549024672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[0]R flush_numbers[0..12]: [92, 93, 93, 88, 92, 92, 68, 70, 91, 70, 84, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549648052Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[0]R generation[0..12]: [56, 57, 57, 55, 56, 56, 44, 46, 56, 46, 52, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549679193Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549703862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[1]R flush_numbers[0..12]: [92, 93, 93, 88, 92, 92, 68, 70, 91, 70, 84, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549726888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[1]R generation[0..12]: [56, 57, 57, 55, 56, 56, 44, 46, 56, 46, 52, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549749773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549774271Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[2]R flush_numbers[0..12]: [92, 93, 93, 88, 92, 92, 68, 70, 91, 70, 84, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549797036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[2]R generation[0..12]: [56, 57, 57, 55, 56, 56, 44, 46, 56, 46, 52, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549819721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549850552Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549872314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549893947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Next flush: 94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549913897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549934076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"":"downstairs","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549955138Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549974436Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"0","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.549995538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"1","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550016579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"client":"2","":"downstairs","session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550038202Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550058903Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d is now active with session: 8d246e64-21b8-4693-9b86-942ed69d209d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550081698Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550100725Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424,"session_id":"8d246e64-21b8-4693-9b86-942ed69d209d"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550120635Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} {"msg":"Activated sub_volume af6760ca-61d0-4e46-ae6d-f0bf05d8ee4d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:01:55.550138149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 507 len 32768 data: 7 7 7 6 6 4 5 5 02/30 Flush 03/30 Write block 112 len 8192 data: 5 7 04/30 Write block 519 len 40960 data: 8 9 9 9 8 9 12 9 8 7 05/30 Flush 06/30 Write block 341 len 20480 data: 4 3 3 3 4 07/30 Write block 130 len 32768 data: 6 5 5 5 3 2 2 3 08/30 Write block 214 len 4096 data: 3 09/30 Flush 10/30 Write block 120 len 8192 data: 6 5 11/30 Flush 12/30 Write block 361 len 24576 data: 5 5 5 6 5 5 13/30 Write block 493 len 24576 data: 4 5 5 6 7 7 14/30 Flush 15/30 Write block 6 len 4096 data: 3 16/30 Write block 329 len 20480 data: 6 6 4 5 6 17/30 Flush 18/30 Write block 490 len 4096 data: 5 19/30 Read block 425 len 40960 20/30 Write block 356 len 28672 data: 6 5 5 5 5 6 6 21/30 Read block 276 len 24576 22/30 Write block 162 len 8192 data: 4 4 23/30 Read block 65 len 36864 24/30 Read block 328 len 24576 25/30 Write block 203 len 8192 data: 6 6 26/30 Read block 123 len 40960 27/30 Write block 2 len 12288 data: 2 2 2 28/30 Read block 555 len 32768 29/30 Read block 349 len 12288 30/30 Write block 542 len 32768 data: 6 5 5 5 4 6 6 8 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 2 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:01:55.579538677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1424} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 56 58 58 92 99 99 F T T <--- 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 56 58 58 92 95 95 F F F <--- 6 120-139 44 58 58 68 97 97 F F F <--- 7 140-159 46 46 46 70 70 70 F F F 8 160-179 56 56 56 91 91 91 F T T <--- 9 180-199 46 46 46 70 70 70 F F F 10 200-219 52 58 58 84 96 96 F T T <--- 11 220-239 57 57 57 93 93 93 F F F 12 240-259 57 57 57 93 93 93 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 56 56 56 91 91 91 F F F 15 300-319 56 56 56 91 91 91 F F F 16 320-339 56 58 58 91 99 99 F F F <--- 17 340-359 56 58 58 91 96 96 F T T <--- 18 360-379 53 58 58 85 98 98 F T T <--- 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 55 55 88 88 88 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 57 57 57 93 93 93 F F F 23 460-479 57 57 57 93 93 93 F F F 24 480-499 56 58 58 92 98 98 F T T <--- 25 500-519 55 58 58 88 95 95 F F F <--- 26 520-539 57 58 58 93 95 95 F F F <--- 27 540-559 55 55 55 88 88 88 F T T <--- 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 58, Max flush: 99 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 59 59 59 100 100 100 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 59 59 59 100 100 100 F F F 11 220-239 57 57 57 93 93 93 F F F 12 240-259 57 57 57 93 93 93 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 56 56 56 91 91 91 F F F 15 300-319 56 56 56 91 91 91 F F F 16 320-339 58 58 58 99 99 99 F F F 17 340-359 59 59 59 100 100 100 F F F 18 360-379 59 59 59 100 100 100 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 55 55 88 88 88 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 57 57 57 93 93 93 F F F 23 460-479 57 57 57 93 93 93 F F F 24 480-499 59 59 59 100 100 100 F F F 25 500-519 58 58 58 95 95 95 F F F 26 520-539 58 58 58 95 95 95 F F F 27 540-559 59 59 59 100 100 100 F F F 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 59, Max flush: 100 Begin loop 30 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.131119747Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:00.149953072Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.150021687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.166052552Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.166450918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.166531808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.166560114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.166599972Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"Crucible 454d302a-4bfc-4d97-8233-6fde346dbe00 has session id: 76605e3f-417e-4e70-97a8-ea75c0a5f658","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.167705173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Upstairs opts: Upstairs UUID: 454d302a-4bfc-4d97-8233-6fde346dbe00, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.167772386Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Crucible stats registered with UUID: 454d302a-4bfc-4d97-8233-6fde346dbe00","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.167808667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:00.16783526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168442388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168572415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168628806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168678534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168730607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"454d302a-4bfc-4d97-8233-6fde346dbe00 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168773992Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168816677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168861445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168901243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168943457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.168989969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.169035799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"ds_connection connected from Ok(127.0.0.1:57162)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.169485677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"ds_connection connected from Ok(127.0.0.1:38476)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.169574131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"ds_connection connected from Ok(127.0.0.1:64911)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.16985223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.171107147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.171224709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.171496916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.171979107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172200173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[0]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 100, 70, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172517229Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[0]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 59, 46, 59, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172563981Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.1726036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[1]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 100, 70, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172640152Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[1]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 59, 46, 59, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172675481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172712965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[2]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 100, 70, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172748736Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[2]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 59, 46, 59, 57]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172783475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172821019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172851519Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172883111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Next flush: 101","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172913241Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.17294334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"":"downstairs","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.172977257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173041283Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"0","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173075711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"1","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173104628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"client":"2","":"downstairs","session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173133405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173159988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"454d302a-4bfc-4d97-8233-6fde346dbe00 is now active with session: 76605e3f-417e-4e70-97a8-ea75c0a5f658","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:02:00.173190498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173224254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432,"session_id":"76605e3f-417e-4e70-97a8-ea75c0a5f658"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173250927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} {"msg":"Activated sub_volume 454d302a-4bfc-4d97-8233-6fde346dbe00","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:00.173274313Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 524 len 40960 data: 10 13 10 9 8 6 6 7 9 9 02/30 Read block 287 len 8192 03/30 Flush 04/30 Write block 504 len 32768 data: 8 6 6 8 8 8 7 7 05/30 Read block 594 len 12288 06/30 Read block 132 len 4096 07/30 Read block 505 len 8192 08/30 Read block 258 len 8192 09/30 Flush 10/30 Write block 445 len 16384 data: 8 8 6 7 11/30 Read block 355 len 4096 12/30 Write block 410 len 12288 data: 3 3 4 13/30 Read block 398 len 32768 14/30 Flush 15/30 Flush 16/30 Write block 162 len 16384 data: 5 5 4 4 17/30 Write block 350 len 28672 data: 4 4 5 6 4 5 7 18/30 Write block 239 len 40960 data: 5 6 5 6 6 6 4 3 3 4 19/30 Read block 193 len 36864 20/30 Write block 456 len 20480 data: 3 5 7 6 7 21/30 Write block 224 len 20480 data: 3 4 6 6 7 22/30 Read block 427 len 36864 23/30 Write block 549 len 4096 data: 9 24/30 Read block 200 len 16384 25/30 Flush 26/30 Write block 284 len 12288 data: 8 7 6 27/30 Read block 391 len 24576 28/30 Read block 100 len 24576 29/30 Write block 239 len 20480 data: 6 7 6 7 7 30/30 Write block 484 len 12288 data: 8 7 5 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 8 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 2 25 25 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:00.204111727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1432} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 59 60 60 100 105 105 F F F <--- 9 180-199 46 46 46 70 70 70 F F F 10 200-219 59 59 59 100 100 100 F F F 11 220-239 57 60 60 93 105 105 F T T <--- 12 240-259 57 60 60 93 105 105 F T T <--- 13 260-279 52 52 52 82 82 82 F F F 14 280-299 56 56 56 91 91 91 F T T <--- 15 300-319 56 56 56 91 91 91 F F F 16 320-339 58 58 58 99 99 99 F F F 17 340-359 59 60 60 100 105 105 F F F <--- 18 360-379 59 59 59 100 100 100 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 55 60 60 88 103 103 F F F <--- 21 420-439 50 50 50 78 78 78 F F F 22 440-459 57 60 60 93 105 105 F F F <--- 23 460-479 57 60 60 93 105 105 F F F <--- 24 480-499 59 59 59 100 100 100 F T T <--- 25 500-519 58 60 60 95 102 102 T F F <--- 26 520-539 60 60 60 101 101 101 F F F 27 540-559 59 60 60 100 105 105 F F F <--- 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 60, Max flush: 105 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 46 46 46 70 70 70 F F F 8 160-179 60 60 60 105 105 105 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 59 59 59 100 100 100 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 52 52 52 82 82 82 F F F 14 280-299 61 61 61 106 106 106 F F F 15 300-319 56 56 56 91 91 91 F F F 16 320-339 58 58 58 99 99 99 F F F 17 340-359 60 60 60 105 105 105 F F F 18 360-379 59 59 59 100 100 100 F F F 19 380-399 55 55 55 88 88 88 F F F 20 400-419 60 60 60 103 103 103 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 60 60 105 105 105 F F F 23 460-479 60 60 60 105 105 105 F F F 24 480-499 61 61 61 106 106 106 F F F 25 500-519 60 60 60 102 102 102 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 31 31 48 48 48 F F F Max gen: 61, Max flush: 106 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.740743746Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.769537286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.770097582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.770161608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.770183962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.770217569Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"Crucible 47de643e-9707-44e8-a2f8-b0bfb3be8a8e has session id: dd923a00-5a61-4eea-9628-93707e7b3f83","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.771732268Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Upstairs opts: Upstairs UUID: 47de643e-9707-44e8-a2f8-b0bfb3be8a8e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.771791645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Crucible stats registered with UUID: 47de643e-9707-44e8-a2f8-b0bfb3be8a8e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.771825762Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:04.771851182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.772412039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.772466066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.7724936Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.772776629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.77281761Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"47de643e-9707-44e8-a2f8-b0bfb3be8a8e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773015129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773047092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773072953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773099325Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773125277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773254321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773283639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"ds_connection connected from Ok(127.0.0.1:40711)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773311554Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"ds_connection connected from Ok(127.0.0.1:48501)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773534655Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"ds_connection connected from Ok(127.0.0.1:62416)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.773621917Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.774816574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.77509309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.775460665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.77551871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.775695368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[0]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 105, 70, 100, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.77655766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[0]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 60, 46, 59, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776593641Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776627458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[1]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 105, 70, 100, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776659321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[1]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 60, 46, 59, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776690312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776723517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[2]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 70, 105, 70, 100, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776754879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[2]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 46, 60, 46, 59, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.77678589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776826501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776854125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776882291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Next flush: 107","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776909264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776935957Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"":"downstairs","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776963992Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.776989602Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"0","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777017728Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"1","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777046054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"client":"2","":"downstairs","session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:02:04.77707471Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777108267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"47de643e-9707-44e8-a2f8-b0bfb3be8a8e is now active with session: dd923a00-5a61-4eea-9628-93707e7b3f83","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777138586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777163836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440,"session_id":"dd923a00-5a61-4eea-9628-93707e7b3f83"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777190589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} {"msg":"Activated sub_volume 47de643e-9707-44e8-a2f8-b0bfb3be8a8e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:04.777212342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 272 len 28672 data: 7 7 6 7 8 8 8 02/30 Read block 427 len 20480 03/30 Read block 461 len 12288 04/30 Flush 05/30 Flush 06/30 Flush 07/30 Write block 347 len 16384 data: 5 4 3 5 08/30 Read block 268 len 32768 09/30 Read block 269 len 24576 10/30 Read block 543 len 28672 11/30 Read block 516 len 12288 12/30 Write block 390 len 40960 data: 6 4 4 5 4 3 3 3 4 7 13/30 Read block 354 len 12288 14/30 Read block 28 len 20480 15/30 Flush 16/30 Write block 592 len 24576 data: 3 3 2 2 3 3 17/30 Read block 571 len 36864 18/30 Read block 512 len 16384 19/30 Read block 125 len 24576 20/30 Write block 342 len 40960 data: 4 4 4 5 5 6 5 4 6 5 21/30 Write block 159 len 24576 data: 3 3 3 6 6 5 22/30 Flush 23/30 Write block 303 len 20480 data: 8 4 4 5 5 24/30 Write block 213 len 12288 data: 4 4 2 25/30 Read block 196 len 20480 26/30 Read block 138 len 12288 27/30 Write block 477 len 36864 data: 3 4 4 5 5 5 7 9 8 28/30 Read block 259 len 12288 29/30 Read block 394 len 20480 30/30 Read block 410 len 12288 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 3 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:04.810790643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1440} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 46 62 62 70 111 111 F F F <--- 8 160-179 60 62 62 105 111 111 F F F <--- 9 180-199 46 46 46 70 70 70 F F F 10 200-219 59 59 59 100 100 100 F T T <--- 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 52 62 62 82 107 107 T F F <--- 14 280-299 61 61 61 106 106 106 F F F 15 300-319 56 56 56 91 91 91 F T T <--- 16 320-339 58 58 58 99 99 99 F F F 17 340-359 60 62 62 105 111 111 F F F <--- 18 360-379 59 59 59 100 100 100 F F F 19 380-399 55 62 62 88 110 110 F F F <--- 20 400-419 60 60 60 103 103 103 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 60 60 105 105 105 F F F 23 460-479 60 60 60 105 105 105 F T T <--- 24 480-499 61 61 61 106 106 106 F T T <--- 25 500-519 60 60 60 102 102 102 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 56 56 56 92 92 92 F F F 29 580-599 31 62 62 48 111 111 F F F <--- Max gen: 62, Max flush: 111 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 F F F 2 040-059 57 57 57 93 93 93 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 62 62 62 111 111 111 F F F 8 160-179 62 62 62 111 111 111 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 63 63 63 112 112 112 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 62 62 62 107 107 107 F F F 14 280-299 61 61 61 106 106 106 F F F 15 300-319 63 63 63 112 112 112 F F F 16 320-339 58 58 58 99 99 99 F F F 17 340-359 62 62 62 111 111 111 F F F 18 360-379 59 59 59 100 100 100 F F F 19 380-399 62 62 62 110 110 110 F F F 20 400-419 60 60 60 103 103 103 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 60 60 105 105 105 F F F 23 460-479 63 63 63 112 112 112 F F F 24 480-499 63 63 63 112 112 112 F F F 25 500-519 60 60 60 102 102 102 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 56 56 56 92 92 92 F F F 29 580-599 62 62 62 111 111 111 F F F Max gen: 63, Max flush: 112 Begin loop 32 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.321508415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.341696684Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.342152674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.342212943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.342234796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.342261498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"Crucible f2fc6907-4bed-4ab2-9e09-77e1e02f09ed has session id: 60667708-a09f-42a1-8f8c-31075ec48ef8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.343552726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Upstairs opts: Upstairs UUID: f2fc6907-4bed-4ab2-9e09-77e1e02f09ed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.343607525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Crucible stats registered with UUID: f2fc6907-4bed-4ab2-9e09-77e1e02f09ed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.343634939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:09.343655249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.344294861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.344386632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.344412774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.344816089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"task":"crutest"} {"msg":"f2fc6907-4bed-4ab2-9e09-77e1e02f09ed active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345049089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345076083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345106312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345128586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345150329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345173595Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345211249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345557352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"ds_connection connected from Ok(127.0.0.1:45508)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345581049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"ds_connection connected from Ok(127.0.0.1:65465)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345853406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"ds_connection connected from Ok(127.0.0.1:38136)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.345930248Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.347359308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.347508412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.347543933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348151181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348186701Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[0]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 111, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348418639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[0]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 62, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348453537Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.34849013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[1]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 111, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348519528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[1]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 62, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348548525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348578955Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[2]R flush_numbers[0..12]: [100, 93, 93, 88, 92, 95, 97, 111, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348608022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[2]R generation[0..12]: [59, 57, 57, 55, 56, 58, 58, 62, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348636388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348666949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348691908Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348717849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Next flush: 113","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348742808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348767367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"":"downstairs","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348793198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348817025Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"0","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348843186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"1","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348872504Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"client":"2","":"downstairs","session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348899077Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348923755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"f2fc6907-4bed-4ab2-9e09-77e1e02f09ed is now active with session: 60667708-a09f-42a1-8f8c-31075ec48ef8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.34895144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"Wait for a query_work_queue command to finish before sending IO Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.348982672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448,"session_id":"60667708-a09f-42a1-8f8c-31075ec48ef8"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.349008433Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} {"msg":"Activated sub_volume f2fc6907-4bed-4ab2-9e09-77e1e02f09ed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:09.349029935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 482 len 16384 data: 6 8 10 9 02/30 Read block 134 len 8192 03/30 Write block 382 len 28672 data: 8 7 8 6 7 6 6 04/30 Read block 403 len 8192 05/30 Read block 521 len 20480 06/30 Read block 269 len 12288 07/30 Write block 148 len 24576 data: 7 5 6 7 5 5 08/30 Write block 484 len 12288 data: 11 10 6 09/30 Write block 300 len 4096 data: 6 10/30 Read block 454 len 28672 11/30 Write block 320 len 8192 data: 5 4 12/30 Flush 13/30 Write block 43 len 24576 data: 4 5 6 6 5 6 14/30 Read block 209 len 24576 15/30 Write block 318 len 20480 data: 5 5 6 5 3 16/30 Read block 289 len 32768 17/30 Read block 218 len 12288 18/30 Write block 302 len 28672 data: 9 9 5 5 6 6 5 19/30 Write block 593 len 24576 data: 4 3 3 4 4 2 20/30 Read block 403 len 4096 21/30 Read block 318 len 8192 22/30 Write block 578 len 32768 data: 4 6 4 4 3 3 3 3 23/30 Read block 457 len 36864 24/30 Flush 25/30 Write block 359 len 36864 data: 6 6 7 7 6 7 6 6 5 26/30 Write block 53 len 16384 data: 5 7 8 7 27/30 Write block 33 len 24576 data: 6 7 7 6 8 7 28/30 Write block 275 len 16384 data: 8 9 9 9 29/30 Read block 447 len 16384 30/30 Write block 473 len 36864 data: 4 4 4 5 4 5 5 6 6 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Write 7 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 4 Sent Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 1 32 Done 28 1 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:09.374663098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1448} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 57 57 57 93 93 93 T F T <--- 2 040-059 64 57 64 114 93 114 T F T <--- 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 62 64 113 111 113 F F F <--- 8 160-179 62 62 62 111 111 111 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 63 63 63 112 112 112 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 62 62 62 107 107 107 T F T <--- 14 280-299 61 61 61 106 106 106 F F F 15 300-319 64 63 64 114 112 114 F F F <--- 16 320-339 64 58 64 114 99 114 F F F <--- 17 340-359 62 62 62 111 111 111 T F T <--- 18 360-379 59 59 59 100 100 100 T F T <--- 19 380-399 64 62 64 113 110 113 F F F <--- 20 400-419 60 60 60 103 103 103 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 60 60 105 105 105 F F F 23 460-479 63 63 63 112 112 112 T F T <--- 24 480-499 64 63 64 113 112 113 T T T <--- 25 500-519 60 60 60 102 102 102 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 64 56 64 114 92 114 F F F <--- 29 580-599 64 62 64 114 111 114 F F F <--- Max gen: 64, Max flush: 114 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 65 65 65 115 115 115 F F F 2 040-059 65 65 65 115 115 115 F F F 3 060-079 55 55 55 88 88 88 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 62 62 62 111 111 111 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 63 63 63 112 112 112 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 65 65 65 115 115 115 F F F 14 280-299 61 61 61 106 106 106 F F F 15 300-319 64 64 64 114 114 114 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 65 65 115 115 115 F F F 19 380-399 64 64 64 113 113 113 F F F 20 400-419 60 60 60 103 103 103 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 60 60 105 105 105 F F F 23 460-479 65 65 65 115 115 115 F F F 24 480-499 65 65 65 115 115 115 F F F 25 500-519 60 60 60 102 102 102 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 64 64 64 114 114 114 F F F 29 580-599 64 64 64 114 114 114 F F F Max gen: 65, Max flush: 115 Begin loop 33 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.904817766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:13.9237269Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.923834483Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.940546982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.941117208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.941214259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.941237936Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.941272494Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"Crucible c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90 has session id: 9dcb4228-cf73-4c61-8895-569b7f039b6e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.942396653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Upstairs opts: Upstairs UUID: c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.942489075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Crucible stats registered with UUID: c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.942518784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:13.942543162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.942990204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943083588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943110802Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94324154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943339353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94336832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943390584Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943414511Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943437527Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943468097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943695906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943762808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44412)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943800373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61894)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94383939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"ds_connection connected from Ok(127.0.0.1:60947)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.943874178Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.945479155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.945668459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.946096283Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.946174878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.946721577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[0]R flush_numbers[0..12]: [100, 115, 115, 88, 92, 95, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947058141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[0]R generation[0..12]: [59, 65, 65, 55, 56, 58, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947099693Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947134081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[1]R flush_numbers[0..12]: [100, 115, 115, 88, 92, 95, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947163599Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[1]R generation[0..12]: [59, 65, 65, 55, 56, 58, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947192266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947222636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[2]R flush_numbers[0..12]: [100, 115, 115, 88, 92, 95, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947251503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[2]R generation[0..12]: [59, 65, 65, 55, 56, 58, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947287624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947324016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947349246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947380047Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Next flush: 116","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947410637Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94743751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"":"downstairs","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947469784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94749341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"Wait for a query_work_queue command to finish before sending IO client":"0","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947536305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"1","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947570392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"client":"2","":"downstairs","session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947596764Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947626102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90 is now active with session: 9dcb4228-cf73-4c61-8895-569b7f039b6e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.94765547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947679317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456,"session_id":"9dcb4228-cf73-4c61-8895-569b7f039b6e"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947704076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} {"msg":"Activated sub_volume c8c61ad8-4048-4c1d-a5a8-2ec3f48d1a90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:13.947725498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 449 len 4096 data: 7 02/30 Read block 489 len 16384 03/30 Read block 279 len 28672 04/30 Write block 27 len 12288 data: 5 6 8 05/30 Read block 597 len 4096 06/30 Read block 361 len 8192 07/30 Read block 520 len 16384 08/30 Flush 09/30 Write block 66 len 40960 data: 9 8 10 7 6 5 7 8 9 10 10/30 Write block 410 len 20480 data: 4 4 5 5 7 11/30 Write block 43 len 4096 data: 5 12/30 Write block 501 len 40960 data: 8 8 7 9 7 7 9 9 9 8 13/30 Write block 541 len 24576 data: 5 7 6 6 6 5 14/30 Read block 254 len 24576 15/30 Read block 520 len 16384 16/30 Read block 368 len 28672 17/30 Read block 231 len 8192 18/30 Read block 443 len 16384 19/30 Write block 110 len 40960 data: 3 3 6 8 5 7 8 7 7 4 20/30 Write block 507 len 20480 data: 10 10 10 9 8 21/30 Write block 46 len 4096 data: 7 22/30 Read block 79 len 12288 23/30 Write block 471 len 32768 data: 5 4 5 5 5 6 5 6 24/30 Read block 304 len 28672 25/30 Read block 383 len 4096 26/30 Read block 144 len 16384 27/30 Read block 360 len 32768 28/30 Write block 573 len 24576 data: 6 6 6 6 6 5 29/30 Read block 428 len 28672 30/30 Read block 569 len 20480 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1007)) Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:13.974698275Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1456} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 65 66 66 115 116 116 F F F <--- 2 040-059 65 65 65 115 115 115 F T T <--- 3 060-079 55 55 55 88 88 88 F T T <--- 4 080-099 56 56 56 92 92 92 F F F 5 100-119 58 58 58 95 95 95 F T T <--- 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 62 62 62 111 111 111 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 63 63 63 112 112 112 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 65 65 65 115 115 115 F F F 14 280-299 61 61 61 106 106 106 F F F 15 300-319 64 64 64 114 114 114 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 65 65 115 115 115 F F F 19 380-399 64 64 64 113 113 113 F F F 20 400-419 60 60 60 103 103 103 F T T <--- 21 420-439 50 50 50 78 78 78 F F F 22 440-459 60 66 66 105 116 116 F F F <--- 23 460-479 65 65 65 115 115 115 F T T <--- 24 480-499 65 65 65 115 115 115 F F F 25 500-519 60 60 60 102 102 102 F T T <--- 26 520-539 60 60 60 101 101 101 F F F 27 540-559 60 60 60 105 105 105 F T T <--- 28 560-579 64 64 64 114 114 114 F T T <--- 29 580-599 64 64 64 114 114 114 F F F Max gen: 66, Max flush: 116 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 66 66 66 116 116 116 F F F 2 040-059 67 67 67 117 117 117 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 67 67 67 117 117 117 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 62 62 62 111 111 111 F F F 9 180-199 46 46 46 70 70 70 F F F 10 200-219 63 63 63 112 112 112 F F F 11 220-239 61 61 61 106 106 106 F F F 12 240-259 61 61 61 106 106 106 F F F 13 260-279 65 65 65 115 115 115 F F F 14 280-299 61 61 61 106 106 106 F F F 15 300-319 64 64 64 114 114 114 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 65 65 115 115 115 F F F 19 380-399 64 64 64 113 113 113 F F F 20 400-419 67 67 67 117 117 117 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 67 67 67 117 117 117 F F F 24 480-499 65 65 65 115 115 115 F F F 25 500-519 67 67 67 117 117 117 F F F 26 520-539 60 60 60 101 101 101 F F F 27 540-559 67 67 67 117 117 117 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 64 64 64 114 114 114 F F F Max gen: 67, Max flush: 117 Begin loop 34 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.52907374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.554178771Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.554756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.554832181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.554865667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55490304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"Crucible c0cc195f-5b21-4ab2-a471-37dbe88d538b has session id: 50a02e3e-9992-42ca-9ab8-48491846d42b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.555925789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Upstairs opts: Upstairs UUID: c0cc195f-5b21-4ab2-a471-37dbe88d538b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55599245Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Crucible stats registered with UUID: c0cc195f-5b21-4ab2-a471-37dbe88d538b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.556031467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:18.55605814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.556427148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55653943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55658016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.556865253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.556906775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"c0cc195f-5b21-4ab2-a471-37dbe88d538b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55701076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557045298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557074466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557105617Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557140346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557168882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"ds_connection connected from Ok(127.0.0.1:57000)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557274661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557500636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"ds_connection connected from Ok(127.0.0.1:40241)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557544212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"ds_connection connected from Ok(127.0.0.1:59413)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.557581967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.55939238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.559801136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.560054376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.560493983Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.560686113Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[0]R flush_numbers[0..12]: [100, 116, 117, 117, 92, 117, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561028208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[0]R generation[0..12]: [59, 66, 67, 67, 56, 67, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561070341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561107314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[1]R flush_numbers[0..12]: [100, 116, 117, 117, 92, 117, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561142002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[1]R generation[0..12]: [59, 66, 67, 67, 56, 67, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561176039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561212151Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[2]R flush_numbers[0..12]: [100, 116, 117, 117, 92, 117, 97, 113, 111, 70, 112, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561246859Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[2]R generation[0..12]: [59, 66, 67, 67, 56, 67, 58, 64, 62, 46, 63, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561280506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561316396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561346105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561376675Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Next flush: 118","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561417666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561447174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"":"downstairs","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561477785Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.56151103Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"0","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561542262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"1","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561573363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"client":"2","":"downstairs","session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561604755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561634354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"c0cc195f-5b21-4ab2-a471-37dbe88d538b is now active with session: 50a02e3e-9992-42ca-9ab8-48491846d42b","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-06T06:02:18.5616677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561705374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464,"session_id":"50a02e3e-9992-42ca-9ab8-48491846d42b"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561735093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} {"msg":"Activated sub_volume c0cc195f-5b21-4ab2-a471-37dbe88d538b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:18.561760172Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 34 len 24576 data: 8 8 7 9 8 7 02/30 Read block 93 len 4096 03/30 Read block 417 len 4096 04/30 Read block 115 len 28672 05/30 Read block 76 len 4096 06/30 Write block 236 len 32768 data: 6 7 6 7 8 7 8 8 07/30 Read block 226 len 12288 08/30 Read block 10 len 32768 09/30 Read block 45 len 24576 10/30 Read block 309 len 4096 11/30 Write block 406 len 28672 data: 7 5 4 3 5 5 6 12/30 Read block 266 len 16384 13/30 Read block 163 len 8192 14/30 Write block 261 len 4096 data: 9 15/30 Flush 16/30 Write block 281 len 36864 data: 10 10 7 9 8 7 5 6 9 17/30 Write block 198 len 28672 data: 6 4 5 4 6 7 7 18/30 Read block 407 len 12288 19/30 Flush 20/30 Write block 275 len 36864 data: 9 10 10 10 8 10 11 11 8 21/30 Write block 223 len 28672 data: 3 4 5 7 7 8 6 22/30 Read block 186 len 28672 23/30 Write block 537 len 24576 data: 8 7 7 5 6 8 24/30 Read block 25 len 8192 25/30 Write block 296 len 16384 data: 6 6 6 6 26/30 Read block 569 len 28672 27/30 Write block 177 len 8192 data: 4 4 28/30 Write block 165 len 12288 data: 5 8 7 29/30 Write block 195 len 20480 data: 6 7 7 7 5 30/30 Read block 130 len 16384 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 30 3 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:18.592386409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1464} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 68 66 68 118 116 118 F T F <--- 2 040-059 67 67 67 117 117 117 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 67 67 67 117 117 117 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 62 62 62 111 111 111 T F T <--- 9 180-199 68 46 68 119 70 119 T F T <--- 10 200-219 68 63 68 119 112 119 F F F <--- 11 220-239 68 61 68 118 106 118 T F T <--- 12 240-259 68 61 68 118 106 118 F F F <--- 13 260-279 68 65 68 118 115 118 T F T <--- 14 280-299 68 61 68 119 106 119 T F T <--- 15 300-319 64 64 64 114 114 114 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 65 65 115 115 115 F F F 19 380-399 64 64 64 113 113 113 F F F 20 400-419 68 67 68 118 117 118 F F F <--- 21 420-439 50 50 50 78 78 78 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 67 67 67 117 117 117 F F F 24 480-499 65 65 65 115 115 115 F F F 25 500-519 67 67 67 117 117 117 F F F 26 520-539 60 60 60 101 101 101 T F T <--- 27 540-559 67 67 67 117 117 117 T F T <--- 28 560-579 67 67 67 117 117 117 F F F 29 580-599 64 64 64 114 114 114 F F F Max gen: 68, Max flush: 119 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 100 100 100 F F F 1 020-039 68 68 68 118 118 118 F F F 2 040-059 67 67 67 117 117 117 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 56 56 56 92 92 92 F F F 5 100-119 67 67 67 117 117 117 F F F 6 120-139 58 58 58 97 97 97 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 69 69 69 120 120 120 F F F 9 180-199 69 69 69 120 120 120 F F F 10 200-219 68 68 68 119 119 119 F F F 11 220-239 69 69 69 120 120 120 F F F 12 240-259 68 68 68 118 118 118 F F F 13 260-279 69 69 69 120 120 120 F F F 14 280-299 69 69 69 120 120 120 F F F 15 300-319 64 64 64 114 114 114 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 65 65 115 115 115 F F F 19 380-399 64 64 64 113 113 113 F F F 20 400-419 68 68 68 118 118 118 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 67 67 67 117 117 117 F F F 24 480-499 65 65 65 115 115 115 F F F 25 500-519 67 67 67 117 117 117 F F F 26 520-539 69 69 69 120 120 120 F F F 27 540-559 69 69 69 120 120 120 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 64 64 64 114 114 114 F F F Max gen: 69, Max flush: 120 Begin loop 35 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.150697728Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:23.166495963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.166578396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.182776581Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.183224024Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.183330474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.183369461Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.183429189Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"Crucible 54ba29e7-b5ed-4d9f-873e-fb15570766c6 has session id: 68120231-e8ca-47de-8aa9-8626505ad2b7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.184365116Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Upstairs opts: Upstairs UUID: 54ba29e7-b5ed-4d9f-873e-fb15570766c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.184582825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Crucible stats registered with UUID: 54ba29e7-b5ed-4d9f-873e-fb15570766c6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.184618225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:23.184642603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185057181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185105366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185133362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185424466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185463473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"54ba29e7-b5ed-4d9f-873e-fb15570766c6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185663588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185692645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185718276Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185743566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185768555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185865767Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185902209Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"ds_connection connected from Ok(127.0.0.1:46267)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.185982026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"ds_connection connected from Ok(127.0.0.1:47933)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.186028418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"ds_connection connected from Ok(127.0.0.1:58631)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.186251288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.187956142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.188040579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.188259841Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.188975223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189021965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[0]R flush_numbers[0..12]: [100, 118, 117, 117, 92, 117, 97, 113, 120, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189051083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[0]R generation[0..12]: [59, 68, 67, 67, 56, 67, 58, 64, 69, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189080711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189111452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[1]R flush_numbers[0..12]: [100, 118, 117, 117, 92, 117, 97, 113, 120, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189140469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[1]R generation[0..12]: [59, 68, 67, 67, 56, 67, 58, 64, 69, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189168625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189205417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[2]R flush_numbers[0..12]: [100, 118, 117, 117, 92, 117, 97, 113, 120, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189234996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[2]R generation[0..12]: [59, 68, 67, 67, 56, 67, 58, 64, 69, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189263442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189293732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.1893184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189344211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.18936886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189393599Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"":"downstairs","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.18941953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189443397Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"0","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189479318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"1","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189506832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"client":"2","":"downstairs","session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189533004Z","hostname":"bmat-EVT22200007-0000ec3a","Wait for a query_work_queue command to finish before sending IO pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189573093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"54ba29e7-b5ed-4d9f-873e-fb15570766c6 is now active with session: 68120231-e8ca-47de-8aa9-8626505ad2b7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189609996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189649043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472,"session_id":"68120231-e8ca-47de-8aa9-8626505ad2b7"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189684553Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} {"msg":"Activated sub_volume 54ba29e7-b5ed-4d9f-873e-fb15570766c6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:23.189716356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 43 len 28672 data: 6 6 7 8 6 7 5 02/30 Read block 17 len 20480 03/30 Read block 523 len 4096 04/30 Write block 477 len 32768 data: 6 7 6 7 7 7 9 12 05/30 Flush 06/30 Write block 170 len 12288 data: 7 8 6 07/30 Read block 51 len 32768 08/30 Write block 511 len 8192 data: 9 5 09/30 Read block 488 len 28672 10/30 Write block 464 len 32768 data: 6 7 5 4 6 5 5 6 11/30 Write block 90 len 28672 data: 8 6 7 6 4 5 3 12/30 Write block 528 len 8192 data: 9 7 13/30 Write block 114 len 28672 data: 6 8 9 8 8 5 7 14/30 Flush 15/30 Write block 8 len 12288 data: 4 5 7 16/30 Write block 508 len 32768 data: 11 11 10 10 6 6 6 5 17/30 Flush 18/30 Write block 108 len 32768 data: 2 2 4 4 7 9 7 9 19/30 Read block 196 len 16384 20/30 Write block 378 len 28672 data: 9 10 10 10 9 8 9 21/30 Read block 117 len 12288 22/30 Flush 23/30 Write block 582 len 40960 data: 4 4 4 4 3 4 4 4 4 3 24/30 Read block 230 len 8192 25/30 Flush 26/30 Write block 1 len 24576 data: 2 3 3 3 3 4 27/30 Read block 275 len 4096 28/30 Write block 308 len 36864 data: 6 5 4 5 6 7 7 8 6 29/30 Write block 479 len 24576 data: 7 8 8 8 10 13 30/30 Write block 19 len 24576 data: 4 4 5 3 3 3 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 3 3 33 Done 3 27 27 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:23.220075515Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1472} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 70 70 100 123 123 F T T <--- 1 020-039 68 68 68 118 118 118 F T T <--- 2 040-059 67 70 70 117 121 121 T F F <--- 3 060-079 67 67 67 117 117 117 F F F 4 080-099 56 70 70 92 122 122 F F F <--- 5 100-119 67 70 70 117 124 124 F F F <--- 6 120-139 58 70 70 97 122 122 F F F <--- 7 140-159 64 64 64 113 113 113 F F F 8 160-179 69 70 70 120 122 122 F F F <--- 9 180-199 69 69 69 120 120 120 F F F 10 200-219 68 68 68 119 119 119 F F F 11 220-239 69 69 69 120 120 120 F F F 12 240-259 68 68 68 118 118 118 F F F 13 260-279 69 69 69 120 120 120 F F F 14 280-299 69 69 69 120 120 120 F F F 15 300-319 64 64 64 114 114 114 F T T <--- 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 65 70 70 115 124 124 F F F <--- 19 380-399 64 70 70 113 124 124 F F F <--- 20 400-419 68 68 68 118 118 118 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 67 70 70 117 122 122 F T T <--- 24 480-499 65 70 70 115 121 121 F T T <--- 25 500-519 67 70 70 117 123 123 F F F <--- 26 520-539 69 70 70 120 122 122 F F F <--- 27 540-559 69 69 69 120 120 120 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 64 70 70 114 125 125 F F F <--- Max gen: 70, Max flush: 125 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 70 70 70 124 124 124 F F F 6 120-139 70 70 70 122 122 122 F F F 7 140-159 64 64 64 113 113 113 F F F 8 160-179 70 70 70 122 122 122 F F F 9 180-199 69 69 69 120 120 120 F F F 10 200-219 68 68 68 119 119 119 F F F 11 220-239 69 69 69 120 120 120 F F F 12 240-259 68 68 68 118 118 118 F F F 13 260-279 69 69 69 120 120 120 F F F 14 280-299 69 69 69 120 120 120 F F F 15 300-319 71 71 71 126 126 126 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 70 70 70 124 124 124 F F F 19 380-399 70 70 70 124 124 124 F F F 20 400-419 68 68 68 118 118 118 F F F 21 420-439 50 50 50 78 78 78 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 71 71 71 126 126 126 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 70 70 70 122 122 122 F F F 27 540-559 69 69 69 120 120 120 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 70 70 70 125 125 125 F F F Max gen: 71, Max flush: 126 Begin loop 36 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.812743716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.835307893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.83585383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.835986011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.836024046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.836079866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"Crucible a015a470-5cb8-4819-94b7-b8f2619d8067 has session id: d476c8e0-bee8-4f03-90f9-6ed7e5542b78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.837383098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Upstairs opts: Upstairs UUID: a015a470-5cb8-4819-94b7-b8f2619d8067, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.837468587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Crucible stats registered with UUID: a015a470-5cb8-4819-94b7-b8f2619d8067","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.837521341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:27.837562662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838064913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838166754Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838199359Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838451456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838486055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"a015a470-5cb8-4819-94b7-b8f2619d8067 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.8386309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838656912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838679687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.83870167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.83879286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.838820996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.83884352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"ds_connection connected from Ok(127.0.0.1:41192)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.839059396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"ds_connection connected from Ok(127.0.0.1:39712)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.839250583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"ds_connection connected from Ok(127.0.0.1:38563)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.839389046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.84139773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.841465584Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.841777169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842443213Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842783275Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[0]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 124, 122, 113, 122, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842851669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[0]R generation[0..12]: [71, 71, 70, 67, 70, 70, 70, 64, 70, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842898312Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842946016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[1]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 124, 122, 113, 122, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.842987488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[1]R generation[0..12]: [71, 71, 70, 67, 70, 70, 70, 64, 70, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843031544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843076483Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[2]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 124, 122, 113, 122, 120, 119, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.84312048Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[2]R generation[0..12]: [71, 71, 70, 67, 70, 70, 70, 64, 70, 69, 68, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843162012Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.84320689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843234816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:02:27.843264945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Next flush: 127","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843312579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843360053Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"":"downstairs","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.84339902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843436393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"0","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843476162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"1","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843516091Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"client":"2","":"downstairs","session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.8435564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843592972Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"a015a470-5cb8-4819-94b7-b8f2619d8067 is now active with session: d476c8e0-bee8-4f03-90f9-6ed7e5542b78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843625497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843659774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480,"session_id":"d476c8e0-bee8-4f03-90f9-6ed7e5542b78"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:27.843693892Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} {"msg":"Activated sub_volume a015a470-5cb8-4819-94b7-b8f2619d8067","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:02:27.843724181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 210 len 20480 data: 3 2 3 5 5 02/30 Read block 23 len 12288 03/30 Read block 359 len 24576 04/30 Read block 477 len 28672 05/30 Flush 06/30 Write block 488 len 8192 data: 5 6 07/30 Write block 434 len 24576 data: 7 7 7 7 6 5 08/30 Read block 564 len 16384 09/30 Read block 481 len 8192 10/30 Write block 381 len 32768 data: 11 10 9 10 7 8 7 7 11/30 Read block 399 len 24576 12/30 Write block 581 len 16384 data: 5 5 5 5 13/30 Read block 339 len 8192 14/30 Read block 229 len 40960 15/30 Write block 119 len 24576 data: 6 8 6 4 4 5 16/30 Read block 188 len 8192 17/30 Flush 18/30 Write block 213 len 4096 data: 6 19/30 Read block 4 len 8192 20/30 Read block 474 len 4096 21/30 Write block 118 len 12288 data: 9 7 9 22/30 Read block 509 len 20480 23/30 Read block 388 len 4096 24/30 Read block 491 len 8192 25/30 Flush 26/30 Write block 121 len 28672 data: 7 5 5 6 5 6 6 27/30 Read block 568 len 40960 28/30 Read block 322 len 8192 29/30 Write block 241 len 32768 data: 8 9 9 7 5 4 4 5 30/30 Write block 153 len 24576 data: 6 5 4 5 4 3 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 3 Done Done Sent false 1002 Acked Read 6 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 8 Done Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 30 33 Done 29 28 0 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:27.873930329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1480} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 72 72 70 129 129 124 F F F <--- 6 120-139 72 72 70 129 129 122 T T F <--- 7 140-159 64 64 64 113 113 113 T T F <--- 8 160-179 70 70 70 122 122 122 F F F 9 180-199 69 69 69 120 120 120 F F F 10 200-219 72 72 68 129 129 119 F F F <--- 11 220-239 69 69 69 120 120 120 F F F 12 240-259 68 68 68 118 118 118 T T F <--- 13 260-279 69 69 69 120 120 120 F F F 14 280-299 69 69 69 120 120 120 F F F 15 300-319 71 71 71 126 126 126 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 70 70 70 124 124 124 F F F 19 380-399 72 72 70 128 128 124 F F F <--- 20 400-419 68 68 68 118 118 118 F F F 21 420-439 72 72 50 128 128 78 F F F <--- 22 440-459 66 66 66 116 116 116 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 71 128 128 126 F F F <--- 25 500-519 70 70 70 123 123 123 F F F 26 520-539 70 70 70 122 122 122 F F F 27 540-559 69 69 69 120 120 120 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 72 72 70 128 128 125 F F F <--- Max gen: 72, Max flush: 129 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 72 72 72 129 129 129 F F F 6 120-139 73 73 73 130 130 130 F F F 7 140-159 73 73 73 130 130 130 F F F 8 160-179 70 70 70 122 122 122 F F F 9 180-199 69 69 69 120 120 120 F F F 10 200-219 72 72 72 129 129 129 F F F 11 220-239 69 69 69 120 120 120 F F F 12 240-259 73 73 73 130 130 130 F F F 13 260-279 69 69 69 120 120 120 F F F 14 280-299 69 69 69 120 120 120 F F F 15 300-319 71 71 71 126 126 126 F F F 16 320-339 64 64 64 114 114 114 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 70 70 70 124 124 124 F F F 19 380-399 72 72 72 128 128 128 F F F 20 400-419 68 68 68 118 118 118 F F F 21 420-439 72 72 72 128 128 128 F F F 22 440-459 66 66 66 116 116 116 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 70 70 70 122 122 122 F F F 27 540-559 69 69 69 120 120 120 F F F 28 560-579 67 67 67 117 117 117 F F F 29 580-599 72 72 72 128 128 128 F F F Max gen: 73, Max flush: 130 Begin loop 37 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.387250967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.409926664Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.410445468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.410519083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.410544163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.410578671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"Crucible e22b8a1b-87c0-4570-b8c4-f01131397b21 has session id: 916f4bd7-18a1-4dc5-89d0-06896116cc68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.411371335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Upstairs opts: Upstairs UUID: e22b8a1b-87c0-4570-b8c4-f01131397b21, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.411451804Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Crucible stats registered with UUID: e22b8a1b-87c0-4570-b8c4-f01131397b21","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.411484158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:32.411509017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.411962742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412048842Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412080575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412382461Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412415676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"e22b8a1b-87c0-4570-b8c4-f01131397b21 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412563197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.41259508Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412623366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412651642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412679246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.412707642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.41274221Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:32.413313257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:32.413413205Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:32.413445498Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.41346654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.413502781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"ds_connection connected from Ok(127.0.0.1:44987)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.413599833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"ds_connection connected from Ok(127.0.0.1:48902)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.413631495Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.415538559Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.415835154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.416189794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:32.416470768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.415355423Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"ds_connection connected from Ok(127.0.0.1:38708)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.415485419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.418451443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[0]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 122, 120, 129, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.41971716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[0]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 70, 69, 72, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419761097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419795565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[1]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 122, 120, 129, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.41982871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[1]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 70, 69, 72, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419859732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419890292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[2]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 122, 120, 129, 120]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.41991968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[2]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 70, 69, 72, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419947605Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.419978336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420003315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420029797Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Next flush: 131","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420054666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420079485Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"":"downstairs","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420105186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.4201376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"0","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420164122Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"1","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420189893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"client":"2","":"downstairs","session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420216456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420240944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"e22b8a1b-87c0-4570-b8c4-f01131397b21 is now active with session: 916f4bd7-18a1-4dc5-89d0-06896116cc68","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420268699Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420292145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488,"session_id":"916f4bd7-18a1-4dc5-89d0-06896116cc68"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420316513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} {"msg":"Activated sub_volume e22b8a1b-87c0-4570-b8c4-f01131397b21","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:42.420337975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 451 len 36864 data: 5 4 4 3 4 4 6 8 7 03/30 Write block 445 len 20480 data: 9 9 7 8 8 04/30 Write block 535 len 36864 data: 6 8 9 8 8 6 7 9 7 05/30 Read block 225 len 8192 06/30 Write block 234 len 12288 data: 6 6 7 07/30 Read block 99 len 28672 08/30 Read block 3 len 40960 09/30 Read block 214 len 32768 10/30 Flush 11/30 Write block 177 len 40960 data: 5 5 3 5 5 6 8 6 7 7 12/30 Write block 373 len 36864 data: 3 3 4 4 6 10 11 11 12 13/30 Read block 423 len 4096 14/30 Read block 50 len 12288 15/30 Read block 420 len 16384 16/30 Read block 297 len 32768 17/30 Write block 379 len 40960 data: 12 12 13 11 10 11 8 9 8 8 18/30 Write block 316 len 12288 data: 7 5 6 19/30 Write block 330 len 36864 data: 7 5 6 7 6 6 6 5 5 20/30 Read block 292 len 20480 21/30 Read block 124 len 20480 22/30 Write block 295 len 24576 data: 7 7 7 7 7 7 23/30 Read block 571 len 24576 24/30 Read block 397 len 20480 25/30 Write block 336 len 16384 data: 7 6 6 5 26/30 Read block 493 len 20480 27/30 Read block 1 len 16384 28/30 Write block 564 len 36864 data: 5 6 6 8 9 9 10 7 6 29/30 Write block 261 len 40960 data: 10 9 7 7 6 7 7 6 5 4 30/30 Read block 305 len 20480 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 9 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Write 3 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 10 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 5 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Write 10 Done Sent Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 2 23 25 Done 29 27 6 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1009)) Some(JobId(1009)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:42.452664517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1488} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 72 72 72 129 129 129 F F F 6 120-139 73 73 73 130 130 130 F F F 7 140-159 73 73 73 130 130 130 F F F 8 160-179 70 70 70 122 122 122 T T F <--- 9 180-199 69 69 69 120 120 120 T T F <--- 10 200-219 72 72 72 129 129 129 F F F 11 220-239 74 74 69 132 132 120 F F T <--- 12 240-259 73 73 73 130 130 130 F F F 13 260-279 69 69 69 120 120 120 T T F <--- 14 280-299 69 69 69 120 120 120 T T F <--- 15 300-319 71 71 71 126 126 126 T T F <--- 16 320-339 64 64 64 114 114 114 T T F <--- 17 340-359 65 65 65 115 115 115 F F F 18 360-379 70 70 70 124 124 124 T T F <--- 19 380-399 72 72 72 128 128 128 T T F <--- 20 400-419 68 68 68 118 118 118 F F F 21 420-439 72 72 72 128 128 128 F F F 22 440-459 74 74 66 132 132 116 F F T <--- 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 74 74 70 132 132 122 F F T <--- 27 540-559 74 74 69 132 132 120 F F T <--- 28 560-579 67 67 67 117 117 117 T T F <--- 29 580-599 72 72 72 128 128 128 F F F Max gen: 74, Max flush: 132 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 72 72 72 129 129 129 F F F 6 120-139 73 73 73 130 130 130 F F F 7 140-159 73 73 73 130 130 130 F F F 8 160-179 75 75 75 133 133 133 F F F 9 180-199 75 75 75 133 133 133 F F F 10 200-219 72 72 72 129 129 129 F F F 11 220-239 74 74 74 132 132 132 F F F 12 240-259 73 73 73 130 130 130 F F F 13 260-279 75 75 75 133 133 133 F F F 14 280-299 75 75 75 133 133 133 F F F 15 300-319 75 75 75 133 133 133 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 65 65 65 115 115 115 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 75 75 75 133 133 133 F F F 20 400-419 68 68 68 118 118 118 F F F 21 420-439 72 72 72 128 128 128 F F F 22 440-459 74 74 74 132 132 132 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 74 74 74 132 132 132 F F F 27 540-559 74 74 74 132 132 132 F F F 28 560-579 75 75 75 133 133 133 F F F 29 580-599 72 72 72 128 128 128 F F F Max gen: 75, Max flush: 133 Begin loop 38 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:56.993330062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:57.011035491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.011113064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.026139026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.026561099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.026630906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.026659563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.026693891Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"Crucible 0c4cd3e9-8a41-4960-b11c-23341673024b has session id: 590d23e0-80a8-4743-b72f-bc38925f9de6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.027776697Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Upstairs opts: Upstairs UUID: 0c4cd3e9-8a41-4960-b11c-23341673024b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.027834682Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Crucible stats registered with UUID: 0c4cd3e9-8a41-4960-b11c-23341673024b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.027862377Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:57.027887516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028260923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028313827Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028341412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028591966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028620052Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"0c4cd3e9-8a41-4960-b11c-23341673024b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028734167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028762292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028788634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028814876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028840657Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.028874824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.029115169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36103)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.029155288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:35704)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.029183954Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:58570)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.02921214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.030673013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.030784783Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.030820304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031489875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031612337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[0]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 133, 133, 129, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031754517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[0]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 75, 75, 72, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031791079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031822662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[1]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 133, 133, 129, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031852671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[1]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 75, 75, 72, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.03188776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031918651Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[2]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 129, 130, 130, 133, 133, 129, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.0319485Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[2]R generation[0..12]: [71, 71, 70, 67, 70, 72, 73, 73, 75, 75, 72, 74]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.031977046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.03200942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.03203455Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032060401Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032084909Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032109387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"":"downstairs","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032135529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032159226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"0","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032184836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"1","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032210497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"client":"2","":"downstairs","session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:02:57.032236649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032268491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"0c4cd3e9-8a41-4960-b11c-23341673024b is now active with session: 590d23e0-80a8-4743-b72f-bc38925f9de6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032296487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.03232355Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496,"session_id":"590d23e0-80a8-4743-b72f-bc38925f9de6"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032348509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} {"msg":"Activated sub_volume 0c4cd3e9-8a41-4960-b11c-23341673024b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:02:57.032369952Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 114 len 12288 data: 8 10 10 02/30 Flush 03/30 Write block 593 len 4096 data: 5 04/30 Flush 05/30 Write block 411 len 16384 data: 6 7 6 8 06/30 Read block 198 len 32768 07/30 Read block 487 len 16384 08/30 Read block 502 len 32768 09/30 Read block 536 len 32768 10/30 Read block 362 len 12288 11/30 Read block 419 len 28672 12/30 Write block 355 len 8192 data: 6 8 13/30 Write block 539 len 16384 data: 9 7 8 10 14/30 Read block 515 len 8192 15/30 Write block 239 len 28672 data: 8 9 9 10 10 8 6 16/30 Write block 245 len 28672 data: 7 5 5 6 3 3 3 17/30 Write block 565 len 40960 data: 7 7 9 10 10 11 8 7 7 7 18/30 Flush 19/30 Write block 265 len 16384 data: 7 8 8 7 20/30 Write block 127 len 4096 data: 7 21/30 Write block 450 len 8192 data: 6 6 22/30 Write block 183 len 24576 data: 9 7 8 8 7 6 23/30 Read block 278 len 8192 24/30 Read block 47 len 24576 25/30 Read block 295 len 4096 26/30 Write block 276 len 8192 data: 11 11 27/30 Read block 214 len 20480 28/30 Write block 206 len 20480 data: 8 6 4 3 4 29/30 Write block 144 len 12288 data: 8 9 9 30/30 Read block 527 len 16384 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 4 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Read 4 Done Done Done false 1007 Acked Read 8 Done Done Done false 1008 Acked Read 8 Done Done Done false 1009 Acked Read 3 Done Done Done false 1010 Acked Read 7 Done Done Done false 1011 Acked Write 2 Done Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Write 3 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 18 0 1 19 Done 8 26 25 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:02:57.058045288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1496} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 76 76 76 134 134 134 F F F 6 120-139 73 73 73 130 130 130 F T T <--- 7 140-159 73 73 73 130 130 130 F T T <--- 8 160-179 75 75 75 133 133 133 F F F 9 180-199 75 75 75 133 133 133 F T T <--- 10 200-219 72 72 72 129 129 129 F T T <--- 11 220-239 74 76 76 132 136 136 F F F <--- 12 240-259 73 76 76 130 136 136 F F F <--- 13 260-279 75 75 75 133 133 133 F T T <--- 14 280-299 75 75 75 133 133 133 F F F 15 300-319 75 75 75 133 133 133 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 65 76 76 115 136 136 T F F <--- 18 360-379 75 75 75 133 133 133 F F F 19 380-399 75 75 75 133 133 133 F F F 20 400-419 68 76 76 118 136 136 T F F <--- 21 420-439 72 72 72 128 128 128 F F F 22 440-459 74 74 74 132 132 132 F T T <--- 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 74 76 76 132 136 136 F F F <--- 27 540-559 74 76 76 132 136 136 F F F <--- 28 560-579 75 76 76 133 136 136 F F F <--- 29 580-599 76 76 76 135 135 135 F F F Max gen: 76, Max flush: 136 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 76 76 76 134 134 134 F F F 6 120-139 77 77 77 137 137 137 F F F 7 140-159 77 77 77 137 137 137 F F F 8 160-179 75 75 75 133 133 133 F F F 9 180-199 77 77 77 137 137 137 F F F 10 200-219 77 77 77 137 137 137 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 76 76 76 136 136 136 F F F 13 260-279 77 77 77 137 137 137 F F F 14 280-299 75 75 75 133 133 133 F F F 15 300-319 75 75 75 133 133 133 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 75 75 75 133 133 133 F F F 20 400-419 76 76 76 136 136 136 F F F 21 420-439 72 72 72 128 128 128 F F F 22 440-459 77 77 77 137 137 137 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F F F 25 500-519 70 70 70 123 123 123 F F F 26 520-539 76 76 76 136 136 136 F F F 27 540-559 76 76 76 136 136 136 F F F 28 560-579 76 76 76 136 136 136 F F F 29 580-599 76 76 76 135 135 135 F F F Max gen: 77, Max flush: 137 Begin loop 39 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.579193383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:01.595884681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.595983987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.611391472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.612047777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.612113527Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.61213516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.612165529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"Crucible 656945db-70cf-495c-b562-fe8bec1467b6 has session id: c231534c-f012-4aa7-b120-2b14b53627a8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613150343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Upstairs opts: Upstairs UUID: 656945db-70cf-495c-b562-fe8bec1467b6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613249609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Crucible stats registered with UUID: 656945db-70cf-495c-b562-fe8bec1467b6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613276923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:01.613312513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613820656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613872127Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.613898129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614119485Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.6141525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"656945db-70cf-495c-b562-fe8bec1467b6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614260934Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614287477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614311735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614335922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614412273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614440108Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614465037Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:36617)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.61455752Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:57038)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614752865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:60839)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.614937259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.61650357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.616552186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.616866817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617293118Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617326384Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[0]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 134, 137, 137, 133, 137, 137, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617507681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[0]R generation[0..12]: [71, 71, 70, 67, 70, 76, 77, 77, 75, 77, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617538753Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617566528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[1]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 134, 137, 137, 133, 137, 137, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617592689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[1]R generation[0..12]: [71, 71, 70, 67, 70, 76, 77, 77, 75, 77, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617617899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617644772Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[2]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 134, 137, 137, 133, 137, 137, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617671054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[2]R generation[0..12]: [71, 71, 70, 67, 70, 76, 77, 77, 75, 77, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617696384Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617723818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617746563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.61777548Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Next flush: 138","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617797864Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617820158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"":"downstairs","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617843374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617865478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"0","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617890357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"1","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617914755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"client":"2","":"downstairs","session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617939383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617962639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"656945db-70cf-495c-b562-fe8bec1467b6 is now active with session: c231534c-f012-4aa7-b120-2b14b53627a8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.617988931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.618011776Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1504,"session_id":"c231534c-f012-4aa7-b120-2b14b53627a8"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.618046886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} {"msg":"Activated sub_volume 656945db-70cf-495c-b562-fe8bec1467b6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:01.618067075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 588 len 12288 data: 5 5 5 02/30 Flush 03/30 Write block 422 len 16384 data: 7 7 7 4 04/30 Read block 176 len 20480 05/30 Read block 464 len 28672 06/30 Write block 275 len 20480 data: 10 12 12 11 9 07/30 Read block 264 len 32768 08/30 Read block 270 len 40960 09/30 Flush 10/30 Write block 317 len 4096 data: 6 11/30 Write block 597 len 12288 data: 5 3 2 12/30 Write block 212 len 32768 data: 4 7 6 3 2 2 2 2 13/30 Read block 445 len 40960 14/30 Read block 591 len 4096 15/30 Write block 246 len 20480 data: 6 6 7 4 4 16/30 Flush 17/30 Write block 512 len 40960 data: 7 7 7 6 5 6 6 9 10 10 18/30 Write block 161 len 28672 data: 4 7 7 6 6 9 8 19/30 Flush 20/30 Write block 533 len 20480 data: 10 7 7 9 10 21/30 Read block 350 len 20480 22/30 Write block 273 len 28672 data: 8 7 11 13 13 12 10 23/30 Write block 484 len 28672 data: 14 11 7 6 6 7 6 24/30 Write block 286 len 20480 data: 8 6 7 10 7 25/30 Read block 126 len 24576 26/30 Read block 306 len 40960 27/30 Write block 283 len 4096 data: 9 28/30 Write block 399 len 32768 data: 8 8 9 10 12 9 10 8 29/30 Write block 151 len 12288 data: 8 6 7 30/30 Write block 104 len 36864 data: 5 5 4 2 3 3 5 5 8 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:01.645224296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1504} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 76 76 76 134 134 134 F T T <--- 6 120-139 77 77 77 137 137 137 F F F 7 140-159 77 77 77 137 137 137 F T T <--- 8 160-179 75 78 78 133 141 141 F F F <--- 9 180-199 77 77 77 137 137 137 F F F 10 200-219 77 78 78 137 140 140 F F F <--- 11 220-239 76 76 76 136 136 136 F F F 12 240-259 76 78 78 136 140 140 F F F <--- 13 260-279 77 78 78 137 139 139 F T T <--- 14 280-299 75 75 75 133 133 133 F T T <--- 15 300-319 75 78 78 133 140 140 F F F <--- 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 75 75 75 133 133 133 F T T <--- 20 400-419 76 76 76 136 136 136 F T T <--- 21 420-439 72 78 78 128 139 139 F F F <--- 22 440-459 77 77 77 137 137 137 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 72 72 72 128 128 128 F T T <--- 25 500-519 70 78 78 123 141 141 F F F <--- 26 520-539 76 78 78 136 141 141 F T T <--- 27 540-559 76 76 76 136 136 136 F F F 28 560-579 76 76 76 136 136 136 F F F 29 580-599 76 78 78 135 140 140 F F F <--- Max gen: 78, Max flush: 141 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 71 71 126 126 126 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F F F 5 100-119 79 79 79 142 142 142 F F F 6 120-139 77 77 77 137 137 137 F F F 7 140-159 79 79 79 142 142 142 F F F 8 160-179 78 78 78 141 141 141 F F F 9 180-199 77 77 77 137 137 137 F F F 10 200-219 78 78 78 140 140 140 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 78 78 78 140 140 140 F F F 13 260-279 79 79 79 142 142 142 F F F 14 280-299 79 79 79 142 142 142 F F F 15 300-319 78 78 78 140 140 140 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 79 79 79 142 142 142 F F F 20 400-419 79 79 79 142 142 142 F F F 21 420-439 78 78 78 139 139 139 F F F 22 440-459 77 77 77 137 137 137 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 79 79 79 142 142 142 F F F 25 500-519 78 78 78 141 141 141 F F F 26 520-539 79 79 79 142 142 142 F F F 27 540-559 76 76 76 136 136 136 F F F 28 560-579 76 76 76 136 136 136 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 79, Max flush: 142 Begin loop 40 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.173522791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:06.190673144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.190750367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.207073749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.207583415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.20765191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.207674584Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.207705736Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"Crucible 6db04a43-1859-4a4f-aeb1-a4da8c9e6553 has session id: 8b35de25-9dcf-48f2-931e-e0e9f4db7baa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.208625671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Upstairs opts: Upstairs UUID: 6db04a43-1859-4a4f-aeb1-a4da8c9e6553, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.2087621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Crucible stats registered with UUID: 6db04a43-1859-4a4f-aeb1-a4da8c9e6553","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.208793883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:06.208816107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209268099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209318488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.20934472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209654041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209688609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"6db04a43-1859-4a4f-aeb1-a4da8c9e6553 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209806692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209833525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209863835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209888674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209913463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.209938191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.210506242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"ds_connection connected from Ok(127.0.0.1:41167)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.21094082Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"ds_connection connected from Ok(127.0.0.1:50048)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.210971551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"ds_connection connected from Ok(127.0.0.1:57643)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.210998484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.21213189Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.212179554Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.212725632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.212857883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.212888032Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[0]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 142, 137, 142, 141, 137, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213616069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[0]R generation[0..12]: [71, 71, 70, 67, 70, 79, 77, 79, 78, 77, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213654966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.21369259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[1]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 142, 137, 142, 141, 137, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213721247Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[1]R generation[0..12]: [71, 71, 70, 67, 70, 79, 77, 79, 78, 77, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.21374842Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213776997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[2]R flush_numbers[0..12]: [126, 126, 121, 117, 122, 142, 137, 142, 141, 137, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213804751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[2]R generation[0..12]: [71, 71, 70, 67, 70, 79, 77, 79, 78, 77, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213831705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213860361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213883437Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213908286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Next flush: 143","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213931321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213954267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"":"downstairs","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.213978544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214000959Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"0","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214038643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"1","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214068231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"client":"2","":"downstairs","session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214093171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214115996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"6db04a43-1859-4a4f-aeb1-a4da8c9e6553 is now active with session: 8b35de25-9dcf-48f2-931e-e0e9f4db7baa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214142027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.21416398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512,"session_id":"8b35de25-9dcf-48f2-931e-e0e9f4db7baa"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214187076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} {"msg":"Activated sub_volume 6db04a43-1859-4a4f-aeb1-a4da8c9e6553","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:06.214228808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 29 len 8192 data: 9 7 02/30 Read block 336 len 12288 03/30 Read block 520 len 32768 04/30 Write block 109 len 24576 data: 4 6 6 9 10 9 05/30 Write block 403 len 28672 data: 13 10 11 9 6 5 4 06/30 Write block 562 len 16384 data: 4 5 6 8 07/30 Write block 291 len 40960 data: 7 7 5 5 8 8 8 8 8 8 08/30 Flush 09/30 Write block 24 len 36864 data: 4 3 3 6 7 10 8 6 7 10/30 Flush 11/30 Write block 245 len 28672 data: 8 7 7 8 5 5 4 12/30 Read block 461 len 24576 13/30 Write block 308 len 28672 data: 7 6 5 6 7 8 8 14/30 Read block 255 len 28672 15/30 Read block 9 len 24576 16/30 Write block 272 len 24576 data: 8 9 8 12 14 14 17/30 Read block 553 len 20480 18/30 Read block 479 len 28672 19/30 Write block 91 len 16384 data: 7 8 7 5 20/30 Write block 498 len 36864 data: 8 8 7 9 9 8 10 8 8 21/30 Write block 150 len 4096 data: 7 22/30 Read block 534 len 8192 23/30 Write block 188 len 8192 data: 7 5 24/30 Read block 554 len 40960 25/30 Read block 313 len 4096 26/30 Read block 390 len 28672 27/30 Write block 90 len 4096 data: 9 28/30 Write block 164 len 12288 data: 7 7 10 29/30 Write block 424 len 32768 data: 8 5 3 5 5 5 6 6 30/30 Write block 397 len 36864 data: 4 5 9 9 10 11 14 11 12 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Sent Done Done false 1001 Acked Read 3 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1009)) Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:06.240978385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1512} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 71 80 80 126 144 144 F F F <--- 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 70 70 70 122 122 122 F T T <--- 5 100-119 79 80 80 142 143 143 F F F <--- 6 120-139 77 77 77 137 137 137 F F F 7 140-159 79 79 79 142 142 142 F T T <--- 8 160-179 78 78 78 141 141 141 F T T <--- 9 180-199 77 77 77 137 137 137 F T T <--- 10 200-219 78 78 78 140 140 140 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 78 78 78 140 140 140 F T T <--- 13 260-279 79 79 79 142 142 142 F T T <--- 14 280-299 79 80 80 142 143 143 F F F <--- 15 300-319 78 80 80 140 143 143 F T T <--- 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 79 79 79 142 142 142 F T T <--- 20 400-419 79 80 80 142 143 143 F T T <--- 21 420-439 78 78 78 139 139 139 F T T <--- 22 440-459 77 77 77 137 137 137 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 79 79 79 142 142 142 F T T <--- 25 500-519 78 78 78 141 141 141 F T T <--- 26 520-539 79 79 79 142 142 142 F F F 27 540-559 76 76 76 136 136 136 F F F 28 560-579 76 80 80 136 143 143 F F F <--- 29 580-599 78 78 78 140 140 140 F F F Max gen: 80, Max flush: 144 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 80 80 80 144 144 144 F F F 2 040-059 70 70 70 121 121 121 F F F 3 060-079 67 67 67 117 117 117 F F F 4 080-099 81 81 81 145 145 145 F F F 5 100-119 80 80 80 143 143 143 F F F 6 120-139 77 77 77 137 137 137 F F F 7 140-159 81 81 81 145 145 145 F F F 8 160-179 81 81 81 145 145 145 F F F 9 180-199 81 81 81 145 145 145 F F F 10 200-219 78 78 78 140 140 140 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 81 81 81 145 145 145 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 80 80 80 143 143 143 F F F 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 75 75 75 133 133 133 F F F 19 380-399 81 81 81 145 145 145 F F F 20 400-419 81 81 81 145 145 145 F F F 21 420-439 81 81 81 145 145 145 F F F 22 440-459 77 77 77 137 137 137 F F F 23 460-479 71 71 71 126 126 126 F F F 24 480-499 81 81 81 145 145 145 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 79 79 79 142 142 142 F F F 27 540-559 76 76 76 136 136 136 F F F 28 560-579 80 80 80 143 143 143 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 81, Max flush: 145 Begin loop 41 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.804382806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.823761417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.824547629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.824871187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.825131321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.825389109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"Crucible b311e0c9-178e-4040-9ee0-bedb19f90aa9 has session id: 1520219d-5f02-41e4-9981-187baa1785be","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.826746027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Upstairs opts: Upstairs UUID: b311e0c9-178e-4040-9ee0-bedb19f90aa9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.826850453Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Crucible stats registered with UUID: b311e0c9-178e-4040-9ee0-bedb19f90aa9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.826889891Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:10.826912265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827210554Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827260302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827283568Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.82768988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.82772554Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"b311e0c9-178e-4040-9ee0-bedb19f90aa9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827847932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827872961Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827893982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827914583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.827987968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.828015001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.828093015Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"ds_connection connected from Ok(127.0.0.1:63943)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.828585958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"ds_connection connected from Ok(127.0.0.1:56832)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.82904308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"ds_connection connected from Ok(127.0.0.1:40415)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.829133428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.82997045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.830229351Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.830296914Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.830783875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.830822441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[0]R flush_numbers[0..12]: [126, 144, 121, 117, 145, 143, 137, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831027256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[0]R generation[0..12]: [71, 80, 70, 67, 81, 80, 77, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831063267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831092564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[1]R flush_numbers[0..12]: [126, 144, 121, 117, 145, 143, 137, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831121281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[1]R generation[0..12]: [71, 80, 70, 67, 81, 80, 77, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831147773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831176951Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[2]R flush_numbers[0..12]: [126, 144, 121, 117, 145, 143, 137, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831204816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[2]R generation[0..12]: [71, 80, 70, 67, 81, 80, 77, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831231408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.83126777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831428307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831459308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Next flush: 146","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831482434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831562201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"":"downstairs","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831641297Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831666928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"0","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-06T06:03:10.831802696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"1","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831887263Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"client":"2","":"downstairs","session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831914526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831937883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"b311e0c9-178e-4040-9ee0-bedb19f90aa9 is now active with session: 1520219d-5f02-41e4-9981-187baa1785be","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831964545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.831986499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520,"session_id":"1520219d-5f02-41e4-9981-187baa1785be"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.832009444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} {"msg":"Activated sub_volume b311e0c9-178e-4040-9ee0-bedb19f90aa9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:10.832030054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 67 len 16384 data: 9 11 8 7 03/30 Read block 142 len 28672 04/30 Read block 365 len 20480 05/30 Write block 118 len 16384 data: 10 8 10 8 06/30 Write block 401 len 16384 data: 11 12 15 12 07/30 Write block 457 len 8192 data: 7 9 08/30 Write block 88 len 32768 data: 11 10 10 8 9 8 6 6 09/30 Write block 297 len 8192 data: 9 9 10/30 Read block 133 len 12288 11/30 Read block 562 len 16384 12/30 Read block 60 len 20480 13/30 Read block 402 len 28672 14/30 Write block 398 len 16384 data: 6 10 10 12 15/30 Write block 38 len 4096 data: 9 16/30 Write block 366 len 40960 data: 7 6 3 2 2 2 2 4 4 5 17/30 Write block 130 len 24576 data: 7 6 6 6 4 3 18/30 Write block 491 len 12288 data: 4 4 5 19/30 Read block 463 len 28672 20/30 Read block 567 len 16384 21/30 Read block 40 len 40960 22/30 Flush 23/30 Write block 465 len 12288 data: 8 6 5 24/30 Flush 25/30 Write block 59 len 20480 data: 6 7 8 7 8 26/30 Write block 579 len 4096 data: 7 27/30 Read block 171 len 4096 28/30 Write block 539 len 32768 data: 10 8 9 11 8 7 7 6 29/30 Write block 435 len 40960 data: 8 8 8 7 6 5 8 8 8 7 30/30 Write block 421 len 4096 data: 7 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Sent false 1002 Acked Read 7 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 29 35 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:10.856726339Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1520} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 82 82 80 147 147 144 F F F <--- 2 040-059 70 70 70 121 121 121 T T F <--- 3 060-079 82 82 67 147 147 117 T T F <--- 4 080-099 82 82 81 147 147 145 F F F <--- 5 100-119 82 82 80 147 147 143 F F F <--- 6 120-139 82 82 77 147 147 137 F F F <--- 7 140-159 81 81 81 145 145 145 F F F 8 160-179 81 81 81 145 145 145 F F F 9 180-199 81 81 81 145 145 145 F F F 10 200-219 78 78 78 140 140 140 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 81 81 81 145 145 145 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 82 82 80 147 147 143 F F F <--- 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 82 82 75 147 147 133 F F F <--- 19 380-399 82 82 81 147 147 145 F F F <--- 20 400-419 82 82 81 147 147 145 F F F <--- 21 420-439 81 81 81 145 145 145 T T F <--- 22 440-459 82 82 77 147 147 137 T T F <--- 23 460-479 82 82 71 148 148 126 F F F <--- 24 480-499 82 82 81 147 147 145 F F F <--- 25 500-519 81 81 81 145 145 145 F F F 26 520-539 79 79 79 142 142 142 T T F <--- 27 540-559 76 76 76 136 136 136 T T F <--- 28 560-579 80 80 80 143 143 143 T T F <--- 29 580-599 78 78 78 140 140 140 F F F Max gen: 82, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 82 82 82 147 147 147 F F F 2 040-059 83 83 83 149 149 149 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 82 82 82 147 147 147 F F F 5 100-119 82 82 82 147 147 147 F F F 6 120-139 82 82 82 147 147 147 F F F 7 140-159 81 81 81 145 145 145 F F F 8 160-179 81 81 81 145 145 145 F F F 9 180-199 81 81 81 145 145 145 F F F 10 200-219 78 78 78 140 140 140 F F F 11 220-239 76 76 76 136 136 136 F F F 12 240-259 81 81 81 145 145 145 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 82 82 82 147 147 147 F F F 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F F F 18 360-379 82 82 82 147 147 147 F F F 19 380-399 82 82 82 147 147 147 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 83 83 83 149 149 149 F F F 22 440-459 83 83 83 149 149 149 F F F 23 460-479 82 82 82 148 148 148 F F F 24 480-499 82 82 82 147 147 147 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 83 83 83 149 149 149 F F F 27 540-559 83 83 83 149 149 149 F F F 28 560-579 83 83 83 149 149 149 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 83, Max flush: 149 Begin loop 42 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.420298863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:15.437836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.437920977Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.452961979Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.453441144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.453512255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.45353525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.453566943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"Crucible dce551f6-51e7-4f36-909f-126deecc2a94 has session id: 5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.454566926Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Upstairs opts: Upstairs UUID: dce551f6-51e7-4f36-909f-126deecc2a94, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.454622445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Crucible stats registered with UUID: dce551f6-51e7-4f36-909f-126deecc2a94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.454649018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:15.454671773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455021212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455107513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455136249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455481831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455514506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"dce551f6-51e7-4f36-909f-126deecc2a94 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455647839Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455675594Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455699661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455723438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455747886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455772906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.455969844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:41537)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.456067727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:54195)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.45637295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:63912)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.456706618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458005201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458249603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458317186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458782124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458818676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[0]R flush_numbers[0..12]: [126, 147, 149, 149, 147, 147, 147, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458946158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[0]R generation[0..12]: [71, 82, 83, 83, 82, 82, 82, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.458980525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459009613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[1]R flush_numbers[0..12]: [126, 147, 149, 149, 147, 147, 147, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459037408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[1]R generation[0..12]: [71, 82, 83, 83, 82, 82, 82, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459064291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459092837Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[2]R flush_numbers[0..12]: [126, 147, 149, 149, 147, 147, 147, 145, 145, 145, 140, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459120311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[2]R generation[0..12]: [71, 82, 83, 83, 82, 82, 82, 81, 81, 81, 78, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459146914Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.45917549Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459198987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459223124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.45924608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459269295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"":"downstairs","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459293513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-06T06:03:15.459329374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"0","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459356678Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"1","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459382459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"client":"2","":"downstairs","session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.4594086Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459433259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"dce551f6-51e7-4f36-909f-126deecc2a94 is now active with session: 5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459460723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459483919Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528,"session_id":"5c8c666b-350e-4a2a-8fdf-f7e5ea0e79a2"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459508377Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} {"msg":"Activated sub_volume dce551f6-51e7-4f36-909f-126deecc2a94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:15.459529679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 91 len 32768 data: 9 10 9 7 7 4 3 3 02/30 Write block 179 len 40960 data: 4 6 6 7 10 8 9 9 8 8 03/30 Write block 241 len 20480 data: 10 11 11 9 9 04/30 Write block 108 len 40960 data: 4 5 7 7 10 11 10 11 11 9 05/30 Read block 424 len 40960 06/30 Read block 161 len 16384 07/30 Flush 08/30 Write block 146 len 12288 data: 10 7 8 09/30 Flush 10/30 Flush 11/30 Write block 288 len 40960 data: 8 11 8 8 8 6 6 9 9 10 12/30 Read block 28 len 32768 13/30 Read block 65 len 4096 14/30 Write block 373 len 40960 data: 5 5 6 5 7 11 13 13 14 12 15/30 Read block 149 len 20480 16/30 Write block 135 len 4096 data: 4 17/30 Write block 436 len 12288 data: 9 9 8 18/30 Read block 530 len 4096 19/30 Read block 312 len 24576 20/30 Read block 483 len 40960 21/30 Read block 168 len 4096 22/30 Flush 23/30 Write block 176 len 32768 data: 5 6 6 5 7 7 8 11 24/30 Flush 25/30 Write block 341 len 8192 data: 5 5 26/30 Write block 218 len 20480 data: 3 3 2 2 3 27/30 Write block 157 len 28672 data: 5 4 4 4 5 8 8 28/30 Write block 364 len 8192 data: 8 7 29/30 Read block 282 len 4096 30/30 Read block 561 len 20480 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:15.486650729Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1528} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 82 82 82 147 147 147 F F F 2 040-059 83 83 83 149 149 149 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 82 84 84 147 150 150 T F F <--- 5 100-119 82 84 84 147 150 150 F F F <--- 6 120-139 82 84 84 147 153 153 F F F <--- 7 140-159 81 84 84 145 151 151 F T T <--- 8 160-179 81 84 84 145 154 154 T T T <--- 9 180-199 81 84 84 145 154 154 T F F <--- 10 200-219 78 78 78 140 140 140 F T T <--- 11 220-239 76 76 76 136 136 136 F T T <--- 12 240-259 81 84 84 145 150 150 F F F <--- 13 260-279 81 81 81 145 145 145 F F F 14 280-299 82 84 84 147 153 153 F F F <--- 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 76 76 76 136 136 136 F T T <--- 18 360-379 82 84 84 147 153 153 F T T <--- 19 380-399 82 84 84 147 153 153 F F F <--- 20 400-419 82 82 82 147 147 147 F F F 21 420-439 83 84 84 149 153 153 F F F <--- 22 440-459 83 83 83 149 149 149 F F F 23 460-479 82 82 82 148 148 148 F F F 24 480-499 82 82 82 147 147 147 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 83 83 83 149 149 149 F F F 27 540-559 83 83 83 149 149 149 F F F 28 560-579 83 83 83 149 149 149 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 84, Max flush: 154 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 82 82 82 147 147 147 F F F 2 040-059 83 83 83 149 149 149 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 84 84 84 150 150 150 F F F 5 100-119 84 84 84 150 150 150 F F F 6 120-139 84 84 84 153 153 153 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 85 85 85 155 155 155 F F F 9 180-199 84 84 84 154 154 154 F F F 10 200-219 85 85 85 155 155 155 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 84 84 84 150 150 150 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 84 84 84 153 153 153 F F F 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 85 85 85 155 155 155 F F F 18 360-379 85 85 85 155 155 155 F F F 19 380-399 84 84 84 153 153 153 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 84 84 153 153 153 F F F 22 440-459 83 83 83 149 149 149 F F F 23 460-479 82 82 82 148 148 148 F F F 24 480-499 82 82 82 147 147 147 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 83 83 83 149 149 149 F F F 27 540-559 83 83 83 149 149 149 F F F 28 560-579 83 83 83 149 149 149 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 85, Max flush: 155 Begin loop 43 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.035574344Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.061516257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.062171129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.062258762Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.062286817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.062327157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"Crucible 10a028b1-4f90-4bd0-88d6-86b4940a3b63 has session id: 22bee2fd-1ecf-49c3-8a12-a891f3a3621f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.063394143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Upstairs opts: Upstairs UUID: 10a028b1-4f90-4bd0-88d6-86b4940a3b63, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.063485463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Crucible stats registered with UUID: 10a028b1-4f90-4bd0-88d6-86b4940a3b63","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.063520021Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:20.063549639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.06415776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064220714Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064253508Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064524092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064559893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"10a028b1-4f90-4bd0-88d6-86b4940a3b63 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064666403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064700601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064906367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064970864Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.064999941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.06502966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.065335033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"ds_connection connected from Ok(127.0.0.1:51242)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.065441593Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58080)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.065483456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"ds_connection connected from Ok(127.0.0.1:36303)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.065523795Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.066769824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.06685875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.06726949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.067680191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.067718717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[0]R flush_numbers[0..12]: [126, 147, 149, 149, 150, 150, 153, 155, 155, 154, 155, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068155899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[0]R generation[0..12]: [71, 82, 83, 83, 84, 84, 84, 85, 85, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068200958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068234715Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[1]R flush_numbers[0..12]: [126, 147, 149, 149, 150, 150, 153, 155, 155, 154, 155, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068266989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[1]R generation[0..12]: [71, 82, 83, 83, 84, 84, 84, 85, 85, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068297739Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068330784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[2]R flush_numbers[0..12]: [126, 147, 149, 149, 150, 150, 153, 155, 155, 154, 155, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068369821Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[2]R generation[0..12]: [71, 82, 83, 83, 84, 84, 84, 85, 85, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068402386Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068435832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068463286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068491441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Next flush: 156","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068518314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068545338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"":"downstairs","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068573804Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068599425Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"0","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068627901Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"1","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068656126Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"client":"2","":"downstairs","session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068685975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068712868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"10a028b1-4f90-4bd0-88d6-86b4940a3b63 is now active with session: 22bee2fd-1ecf-49c3-8a12-a891f3a3621f","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:03:20.068743198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068780382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536,"session_id":"22bee2fd-1ecf-49c3-8a12-a891f3a3621f"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068807255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} {"msg":"Activated sub_volume 10a028b1-4f90-4bd0-88d6-86b4940a3b63","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:20.068834619Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 471 len 32768 data: 7 5 6 6 6 7 7 8 02/30 Write block 389 len 8192 data: 5 7 03/30 Read block 21 len 20480 04/30 Write block 33 len 12288 data: 7 9 9 05/30 Read block 240 len 4096 06/30 Read block 285 len 8192 07/30 Read block 486 len 20480 08/30 Read block 269 len 8192 09/30 Write block 552 len 28672 data: 8 6 6 6 5 4 4 10/30 Read block 343 len 32768 11/30 Read block 282 len 4096 12/30 Read block 341 len 8192 13/30 Read block 263 len 28672 14/30 Write block 521 len 20480 data: 11 10 9 11 14 15/30 Write block 472 len 28672 data: 6 7 7 7 8 8 9 16/30 Read block 287 len 4096 17/30 Read block 117 len 40960 18/30 Flush 19/30 Write block 196 len 24576 data: 8 8 8 6 6 5 20/30 Flush 21/30 Write block 117 len 36864 data: 10 11 9 11 9 6 6 7 6 22/30 Read block 139 len 16384 23/30 Read block 597 len 8192 24/30 Read block 82 len 12288 25/30 Read block 182 len 36864 26/30 Write block 252 len 12288 data: 3 5 5 27/30 Read block 157 len 12288 28/30 Write block 381 len 32768 data: 15 13 11 12 9 10 9 9 29/30 Read block 148 len 16384 30/30 Write block 87 len 36864 data: 9 12 11 11 10 11 10 8 8 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Read 2 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 29 29 7 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:20.101955258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1536} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 86 86 82 156 156 147 F F T <--- 2 040-059 83 83 83 149 149 149 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 84 84 84 150 150 150 T T F <--- 5 100-119 84 84 84 150 150 150 T T F <--- 6 120-139 84 84 84 153 153 153 T T F <--- 7 140-159 85 85 85 155 155 155 F F F 8 160-179 85 85 85 155 155 155 F F F 9 180-199 86 86 84 157 157 154 F F F <--- 10 200-219 86 86 85 157 157 155 F F F <--- 11 220-239 85 85 85 155 155 155 F F F 12 240-259 84 84 84 150 150 150 T T F <--- 13 260-279 81 81 81 145 145 145 F F F 14 280-299 84 84 84 153 153 153 F F F 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 85 85 85 155 155 155 F F F 18 360-379 85 85 85 155 155 155 F F F 19 380-399 86 86 84 156 156 153 T T T <--- 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 84 84 153 153 153 F F F 22 440-459 83 83 83 149 149 149 F F F 23 460-479 86 86 82 156 156 148 F F T <--- 24 480-499 82 82 82 147 147 147 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 86 86 83 156 156 149 F F F <--- 27 540-559 86 86 83 156 156 149 F F F <--- 28 560-579 83 83 83 149 149 149 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 86, Max flush: 157 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 F F F 1 020-039 86 86 86 156 156 156 F F F 2 040-059 83 83 83 149 149 149 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 87 87 87 158 158 158 F F F 5 100-119 87 87 87 158 158 158 F F F 6 120-139 87 87 87 158 158 158 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 85 85 85 155 155 155 F F F 9 180-199 86 86 86 157 157 157 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 87 87 87 158 158 158 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 84 84 84 153 153 153 F F F 15 300-319 81 81 81 145 145 145 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 85 85 85 155 155 155 F F F 18 360-379 85 85 85 155 155 155 F F F 19 380-399 87 87 87 158 158 158 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 84 84 153 153 153 F F F 22 440-459 83 83 83 149 149 149 F F F 23 460-479 86 86 86 156 156 156 F F F 24 480-499 82 82 82 147 147 147 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 86 86 86 156 156 156 F F F 27 540-559 86 86 86 156 156 156 F F F 28 560-579 83 83 83 149 149 149 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 87, Max flush: 158 Begin loop 44 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.653237587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.673284006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.673789954Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.673858219Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.673879882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.673910372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"Crucible a480fdbc-547a-4452-ab09-ab2fe796688e has session id: 5678916f-28d9-46a0-8b6f-c9546e52f5e4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.67501922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Upstairs opts: Upstairs UUID: a480fdbc-547a-4452-ab09-ab2fe796688e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675116382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Crucible stats registered with UUID: a480fdbc-547a-4452-ab09-ab2fe796688e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675159597Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:24.675196089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675538575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675619445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675644143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.675901351Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.6759364Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"a480fdbc-547a-4452-ab09-ab2fe796688e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676040626Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676067098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676090123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676112768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676134842Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676158579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:55370)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676340478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676368914Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56033)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676394083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:49233)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.676607575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.678189156Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.678618182Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.678989986Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679181293Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679573477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[0]R flush_numbers[0..12]: [126, 156, 149, 149, 158, 158, 158, 155, 155, 157, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679734335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[0]R generation[0..12]: [71, 86, 83, 83, 87, 87, 87, 85, 85, 86, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679770045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679800876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[1]R flush_numbers[0..12]: [126, 156, 149, 149, 158, 158, 158, 155, 155, 157, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679828901Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[1]R generation[0..12]: [71, 86, 83, 83, 87, 87, 87, 85, 85, 86, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679855544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.67988411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[2]R flush_numbers[0..12]: [126, 156, 149, 149, 158, 158, 158, 155, 155, 157, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679911555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[2]R generation[0..12]: [71, 86, 83, 83, 87, 87, 87, 85, 85, 86, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679937776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679965932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.679988967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680013105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Next flush: 159","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:03:24.68003616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680072402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"":"downstairs","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.68009667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680122872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"0","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680147049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"1","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680170916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"client":"2","":"downstairs","session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680195445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.68021813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"a480fdbc-547a-4452-ab09-ab2fe796688e is now active with session: 5678916f-28d9-46a0-8b6f-c9546e52f5e4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680244061Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680265683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544,"session_id":"5678916f-28d9-46a0-8b6f-c9546e52f5e4"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680288478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} {"msg":"Activated sub_volume a480fdbc-547a-4452-ab09-ab2fe796688e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:24.680308428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 475 len 40960 data: 8 9 9 10 8 9 9 9 11 15 03/30 Read block 94 len 12288 04/30 Write block 346 len 16384 data: 6 7 6 5 05/30 Write block 557 len 12288 data: 5 5 5 06/30 Read block 345 len 40960 07/30 Flush 08/30 Write block 193 len 16384 data: 5 7 7 9 09/30 Flush 10/30 Flush 11/30 Write block 371 len 20480 data: 3 3 6 6 7 12/30 Write block 383 len 12288 data: 12 13 10 13/30 Read block 52 len 20480 14/30 Read block 425 len 24576 15/30 Write block 110 len 28672 data: 8 8 11 12 11 12 12 16/30 Write block 453 len 24576 data: 5 4 5 5 8 10 17/30 Write block 302 len 8192 data: 10 10 18/30 Read block 274 len 32768 19/30 Write block 359 len 4096 data: 7 20/30 Write block 53 len 12288 data: 6 8 9 21/30 Read block 514 len 40960 22/30 Read block 77 len 40960 23/30 Read block 266 len 20480 24/30 Write block 565 len 16384 data: 9 8 10 11 25/30 Read block 512 len 32768 26/30 Flush 27/30 Write block 12 len 24576 data: 10 10 7 7 5 5 28/30 Write block 164 len 4096 data: 8 29/30 Write block 90 len 4096 data: 12 30/30 Read block 162 len 24576 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 3 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 29 0 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1000)) Some(JobId(1025)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:24.705884308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1544} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 126 126 126 T F T <--- 1 020-039 86 86 86 156 156 156 F F F 2 040-059 88 83 88 163 149 163 F F F <--- 3 060-079 83 83 83 149 149 149 F F F 4 080-099 87 87 87 158 158 158 T F T <--- 5 100-119 88 87 88 163 158 163 F F F <--- 6 120-139 87 87 87 158 158 158 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 85 85 85 155 155 155 T F T <--- 9 180-199 88 86 88 161 157 161 F F F <--- 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 87 87 87 158 158 158 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 84 84 84 153 153 153 F F F 15 300-319 88 81 88 163 145 163 F F F <--- 16 320-339 75 75 75 133 133 133 F F F 17 340-359 88 85 88 163 155 163 F F F <--- 18 360-379 88 85 88 163 155 163 F F F <--- 19 380-399 88 87 88 163 158 163 F F F <--- 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 84 84 153 153 153 F F F 22 440-459 88 83 88 163 149 163 F F F <--- 23 460-479 88 86 88 160 156 160 F F F <--- 24 480-499 88 82 88 160 147 160 F F F <--- 25 500-519 81 81 81 145 145 145 F F F 26 520-539 86 86 86 156 156 156 F F F 27 540-559 88 86 88 160 156 160 F F F <--- 28 560-579 88 83 88 163 149 163 F F F <--- 29 580-599 78 78 78 140 140 140 F F F Max gen: 88, Max flush: 163 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 164 164 164 F F F 1 020-039 86 86 86 156 156 156 F F F 2 040-059 88 88 88 163 163 163 F F F 3 060-079 83 83 83 149 149 149 F F F 4 080-099 89 89 89 164 164 164 F F F 5 100-119 88 88 88 163 163 163 F F F 6 120-139 87 87 87 158 158 158 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 89 89 89 164 164 164 F F F 9 180-199 88 88 88 161 161 161 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 87 87 87 158 158 158 F F F 13 260-279 81 81 81 145 145 145 F F F 14 280-299 84 84 84 153 153 153 F F F 15 300-319 88 88 88 163 163 163 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 88 88 88 163 163 163 F F F 18 360-379 88 88 88 163 163 163 F F F 19 380-399 88 88 88 163 163 163 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 84 84 153 153 153 F F F 22 440-459 88 88 88 163 163 163 F F F 23 460-479 88 88 88 160 160 160 F F F 24 480-499 88 88 88 160 160 160 F F F 25 500-519 81 81 81 145 145 145 F F F 26 520-539 86 86 86 156 156 156 F F F 27 540-559 88 88 88 160 160 160 F F F 28 560-579 88 88 88 163 163 163 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 89, Max flush: 164 Begin loop 45 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.285794061Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:29.305150809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.305233722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.322290041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.322991305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.323075781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.32310059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.323137463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"Crucible c83cca55-3f92-433a-807b-e19acc9e1d7c has session id: 0d0a29a5-9a26-491f-aa4d-24475a49a877","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324424944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Upstairs opts: Upstairs UUID: c83cca55-3f92-433a-807b-e19acc9e1d7c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324491735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Crucible stats registered with UUID: c83cca55-3f92-433a-807b-e19acc9e1d7c","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324522356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:29.324548588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324911895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324967825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.324997123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325348897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325395028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"c83cca55-3f92-433a-807b-e19acc9e1d7c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325530696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325561256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325589281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325617206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325644731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.32568005Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.325714629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"ds_connection connected from Ok(127.0.0.1:34282)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.326074038Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"ds_connection connected from Ok(127.0.0.1:39084)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.326205598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"ds_connection connected from Ok(127.0.0.1:43353)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.326240887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.32853805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.328618239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.328736522Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329243332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329547021Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[0]R flush_numbers[0..12]: [164, 156, 163, 149, 164, 163, 158, 155, 164, 161, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329720434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[0]R generation[0..12]: [89, 86, 88, 83, 89, 88, 87, 85, 89, 88, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329755603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329788668Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[1]R flush_numbers[0..12]: [164, 156, 163, 149, 164, 163, 158, 155, 164, 161, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329820431Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[1]R generation[0..12]: [89, 86, 88, 83, 89, 88, 87, 85, 89, 88, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329850711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329883225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[2]R flush_numbers[0..12]: [164, 156, 163, 149, 164, 163, 158, 155, 164, 161, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329920478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[2]R generation[0..12]: [89, 86, 88, 83, 89, 88, 87, 85, 89, 88, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329951439Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.329984064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330010797Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330039092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Next flush: 165","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330065535Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330091957Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"":"downstairs","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330119722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330145031Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"0","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330172646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"1","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:03:29.33020019Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"client":"2","":"downstairs","session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330237714Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330264177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"c83cca55-3f92-433a-807b-e19acc9e1d7c is now active with session: 0d0a29a5-9a26-491f-aa4d-24475a49a877","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330294066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330319135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552,"session_id":"0d0a29a5-9a26-491f-aa4d-24475a49a877"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330348844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} {"msg":"Activated sub_volume c83cca55-3f92-433a-807b-e19acc9e1d7c","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:29.330371879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 130 len 8192 data: 8 7 02/30 Write block 438 len 16384 data: 9 7 6 9 03/30 Write block 285 len 28672 data: 9 9 7 9 12 9 9 04/30 Read block 310 len 12288 05/30 Write block 537 len 20480 data: 11 9 11 9 10 06/30 Read block 46 len 32768 07/30 Write block 75 len 32768 data: 11 11 11 10 9 9 11 13 08/30 Read block 370 len 8192 09/30 Write block 255 len 40960 data: 5 7 7 8 10 8 11 10 8 8 10/30 Read block 554 len 20480 11/30 Read block 563 len 40960 12/30 Read block 192 len 32768 13/30 Read block 452 len 16384 14/30 Read block 19 len 8192 15/30 Write block 74 len 16384 data: 10 12 12 12 16/30 Flush 17/30 Write block 447 len 32768 data: 8 9 9 7 7 5 6 5 18/30 Read block 553 len 32768 19/30 Write block 277 len 40960 data: 15 13 11 11 12 12 10 10 10 10 20/30 Write block 496 len 36864 data: 7 8 9 9 8 10 10 9 11 21/30 Read block 308 len 4096 22/30 Read block 131 len 32768 23/30 Write block 346 len 12288 data: 7 8 7 24/30 Read block 120 len 20480 25/30 Read block 114 len 32768 26/30 Write block 182 len 8192 data: 9 12 27/30 Read block 193 len 40960 28/30 Read block 87 len 4096 29/30 Write block 428 len 16384 data: 6 6 7 7 30/30 Write block 32 len 28672 data: 8 8 10 10 8 10 10 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 2 Done Done Done false 1008 Acked Write 10 Done Done Done false 1009 Acked Read 5 Done Done Done false 1010 Acked Read 10 Done Done Done false 1011 Acked Read 8 Done Done Done false 1012 Acked Read 4 Done Done Done false 1013 Acked Read 2 Done Done Done false 1014 Acked Write 4 Done Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 15 2 2 19 Done 15 28 28 71 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:29.360955301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1552} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 164 164 164 F F F 1 020-039 86 86 86 156 156 156 F T T <--- 2 040-059 88 88 88 163 163 163 F F F 3 060-079 83 90 90 149 165 165 T F F <--- 4 080-099 89 90 90 164 165 165 T F F <--- 5 100-119 88 88 88 163 163 163 F F F 6 120-139 87 90 90 158 165 165 T F F <--- 7 140-159 85 85 85 155 155 155 F F F 8 160-179 89 89 89 164 164 164 F F F 9 180-199 88 88 88 161 161 161 F T T <--- 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 87 90 90 158 165 165 T F F <--- 13 260-279 81 90 90 145 165 165 T T T <--- 14 280-299 84 90 90 153 165 165 T T T <--- 15 300-319 88 88 88 163 163 163 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 88 88 88 163 163 163 F T T <--- 18 360-379 88 88 88 163 163 163 F F F 19 380-399 88 88 88 163 163 163 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 84 90 90 153 165 165 T T T <--- 22 440-459 88 90 90 163 165 165 T T T <--- 23 460-479 88 88 88 160 160 160 F F F 24 480-499 88 88 88 160 160 160 F T T <--- 25 500-519 81 81 81 145 145 145 F T T <--- 26 520-539 86 90 90 156 165 165 T F F <--- 27 540-559 88 90 90 160 165 165 T F F <--- 28 560-579 88 88 88 163 163 163 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 90, Max flush: 165 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 164 164 164 F F F 1 020-039 91 91 91 166 166 166 F F F 2 040-059 88 88 88 163 163 163 F F F 3 060-079 90 90 90 165 165 165 F F F 4 080-099 90 90 90 165 165 165 F F F 5 100-119 88 88 88 163 163 163 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 89 89 89 164 164 164 F F F 9 180-199 91 91 91 166 166 166 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 90 90 90 165 165 165 F F F 13 260-279 91 91 91 166 166 166 F F F 14 280-299 91 91 91 166 166 166 F F F 15 300-319 88 88 88 163 163 163 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 91 91 166 166 166 F F F 18 360-379 88 88 88 163 163 163 F F F 19 380-399 88 88 88 163 163 163 F F F 20 400-419 82 82 82 147 147 147 F F F 21 420-439 91 91 91 166 166 166 F F F 22 440-459 91 91 91 166 166 166 F F F 23 460-479 88 88 88 160 160 160 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 91 91 91 166 166 166 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 90 90 90 165 165 165 F F F 28 560-579 88 88 88 163 163 163 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 91, Max flush: 166 Begin loop 46 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.94661071Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.967355227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.967993586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.968069196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.968092281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.968126358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"Crucible 63252c6d-9314-419c-bffa-abf32b09e69b has session id: d77bd707-eafd-4de0-bcf8-394b5453b0f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.968938391Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Upstairs opts: Upstairs UUID: 63252c6d-9314-419c-bffa-abf32b09e69b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.969076814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Crucible stats registered with UUID: 63252c6d-9314-419c-bffa-abf32b09e69b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.96911042Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:33.969134678Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.969554677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.969656097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.96969308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.969932863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.969967411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"63252c6d-9314-419c-bffa-abf32b09e69b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970103901Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970130944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970155352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970180161Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970203998Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970444192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970558839Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:52819)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970591283Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:56471)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970618576Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:38959)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.970672333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.97206469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.972149097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.97277391Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.972866863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.972907924Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[0]R flush_numbers[0..12]: [164, 166, 163, 165, 165, 163, 165, 155, 164, 166, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973596423Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[0]R generation[0..12]: [89, 91, 88, 90, 90, 88, 90, 85, 89, 91, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973629809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973658575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[1]R flush_numbers[0..12]: [164, 166, 163, 165, 165, 163, 165, 155, 164, 166, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.97368628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[1]R generation[0..12]: [89, 91, 88, 90, 90, 88, 90, 85, 89, 91, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973712412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973740487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[2]R flush_numbers[0..12]: [164, 166, 163, 165, 165, 163, 165, 155, 164, 166, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.97376746Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[2]R generation[0..12]: [89, 91, 88, 90, 90, 88, 90, 85, 89, 91, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973793582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973821577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973844723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.97386881Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Next flush: 167","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973891475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973914561Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"":"downstairs","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973944149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973966563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"0","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.973990791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"1","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974014899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"client":"2","":"downstairs","session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974040048Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974063334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"63252c6d-9314-419c-bffa-abf32b09e69b is now active with session: d77bd707-eafd-4de0-bcf8-394b5453b0f5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974089255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974111309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560,"session_id":"d77bd707-eafd-4de0-bcf8-394b5453b0f5"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974134414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} {"msg":"Activated sub_volume 63252c6d-9314-419c-bffa-abf32b09e69b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:33.974154484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 28 len 32768 data: 8 11 9 7 9 9 11 11 02/30 Read block 310 len 24576 03/30 Flush 04/30 Write block 189 len 28672 data: 6 5 6 5 6 8 8 05/30 Read block 250 len 28672 06/30 Write block 160 len 28672 data: 5 6 9 9 9 8 11 07/30 Read block 174 len 16384 08/30 Read block 560 len 28672 09/30 Read block 129 len 8192 10/30 Read block 10 len 24576 11/30 Read block 482 len 8192 12/30 Write block 566 len 16384 data: 9 11 12 11 13/30 Read block 45 len 36864 14/30 Read block 351 len 32768 15/30 Flush 16/30 Write block 450 len 32768 data: 8 8 6 7 6 6 6 9 17/30 Read block 229 len 8192 18/30 Flush 19/30 Write block 256 len 32768 data: 8 8 9 11 9 12 11 9 20/30 Write block 408 len 4096 data: 6 21/30 Write block 507 len 4096 data: 11 22/30 Write block 242 len 4096 data: 12 23/30 Read block 333 len 4096 24/30 Read block 73 len 4096 25/30 Write block 10 len 32768 data: 8 10 11 11 8 8 6 6 26/30 Write block 18 len 16384 data: 4 5 5 6 27/30 Write block 548 len 24576 data: 7 10 8 7 9 7 28/30 Write block 29 len 4096 data: 12 29/30 Write block 415 len 12288 data: 6 6 6 30/30 Write block 451 len 20480 data: 9 7 8 7 7 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Read 6 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 8 Sent Sent Sent false 1025 Acked Write 4 Sent Sent Sent false 1026 Acked Write 6 Sent Sent Sent false 1027 Acked Write 1 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 6 6 30 42 Done 24 24 0 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:33.998221227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1560} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 89 89 89 164 164 164 T T F <--- 1 020-039 92 92 91 167 167 166 T T F <--- 2 040-059 88 88 88 163 163 163 F F F 3 060-079 90 90 90 165 165 165 F F F 4 080-099 90 90 90 165 165 165 F F F 5 100-119 88 88 88 163 163 163 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 92 92 89 168 168 164 F F F <--- 9 180-199 92 92 91 168 168 166 F F F <--- 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 90 90 90 165 165 165 T T F <--- 13 260-279 91 91 91 166 166 166 T T F <--- 14 280-299 91 91 91 166 166 166 F F F 15 300-319 88 88 88 163 163 163 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 91 91 166 166 166 F F F 18 360-379 88 88 88 163 163 163 F F F 19 380-399 88 88 88 163 163 163 F F F 20 400-419 82 82 82 147 147 147 T T F <--- 21 420-439 91 91 91 166 166 166 F F F 22 440-459 92 92 91 169 169 166 T T F <--- 23 460-479 88 88 88 160 160 160 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 91 91 91 166 166 166 T T F <--- 26 520-539 90 90 90 165 165 165 F F F 27 540-559 90 90 90 165 165 165 T T F <--- 28 560-579 92 92 88 168 168 163 F F F <--- 29 580-599 78 78 78 140 140 140 F F F Max gen: 92, Max flush: 169 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 170 170 170 F F F 1 020-039 93 93 93 170 170 170 F F F 2 040-059 88 88 88 163 163 163 F F F 3 060-079 90 90 90 165 165 165 F F F 4 080-099 90 90 90 165 165 165 F F F 5 100-119 88 88 88 163 163 163 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 85 85 85 155 155 155 F F F 8 160-179 92 92 92 168 168 168 F F F 9 180-199 92 92 92 168 168 168 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 85 85 85 155 155 155 F F F 12 240-259 93 93 93 170 170 170 F F F 13 260-279 93 93 93 170 170 170 F F F 14 280-299 91 91 91 166 166 166 F F F 15 300-319 88 88 88 163 163 163 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 91 91 166 166 166 F F F 18 360-379 88 88 88 163 163 163 F F F 19 380-399 88 88 88 163 163 163 F F F 20 400-419 93 93 93 170 170 170 F F F 21 420-439 91 91 91 166 166 166 F F F 22 440-459 93 93 93 170 170 170 F F F 23 460-479 88 88 88 160 160 160 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 93 93 93 170 170 170 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 93 93 93 170 170 170 F F F 28 560-579 92 92 92 168 168 168 F F F 29 580-599 78 78 78 140 140 140 F F F Max gen: 93, Max flush: 170 Begin loop 47 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.579347289Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.599397836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.600102536Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.600174749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.600196361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.600231912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"Crucible 19a6dffc-9b9f-470c-9af9-54f119f58611 has session id: 4c557c37-3f16-4ff0-b790-a6c35501dec5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.601205162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Upstairs opts: Upstairs UUID: 19a6dffc-9b9f-470c-9af9-54f119f58611, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.601364817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Crucible stats registered with UUID: 19a6dffc-9b9f-470c-9af9-54f119f58611","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.60140148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:38.601426269Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.601738845Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.601784646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.601812601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602221127Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602251317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"19a6dffc-9b9f-470c-9af9-54f119f58611 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602380231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602407505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602431923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602456482Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602480228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602504987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602536249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"ds_connection connected from Ok(127.0.0.1:35184)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602893805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"ds_connection connected from Ok(127.0.0.1:48242)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.602985786Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"ds_connection connected from Ok(127.0.0.1:47006)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.60301845Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.60455882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.604601254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.604793724Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.605426222Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.605746905Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[0]R flush_numbers[0..12]: [170, 170, 163, 165, 165, 163, 165, 155, 168, 168, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.60594232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[0]R generation[0..12]: [93, 93, 88, 90, 90, 88, 90, 85, 92, 92, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.605977139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606007449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[1]R flush_numbers[0..12]: [170, 170, 163, 165, 165, 163, 165, 155, 168, 168, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606036657Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[1]R generation[0..12]: [93, 93, 88, 90, 90, 88, 90, 85, 92, 92, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606064542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.60610437Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[2]R flush_numbers[0..12]: [170, 170, 163, 165, 165, 163, 165, 155, 168, 168, 157, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606135482Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[2]R generation[0..12]: [93, 93, 88, 90, 90, 88, 90, 85, 92, 92, 86, 85]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606163837Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606194147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606218666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606244266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Next flush: 171","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606268544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606292792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"":"downstairs","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606318453Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606341829Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"0","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606367499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"1","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606397719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"client":"2","":"downstairs","session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606423991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606448509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"19a6dffc-9b9f-470c-9af9-54f119f58611 is now active with session: 4c557c37-3f16-4ff0-b790-a6c35501dec5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606476094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606499149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568,"session_id":"4c557c37-3f16-4ff0-b790-a6c35501dec5"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606527144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} {"msg":"Activated sub_volume 19a6dffc-9b9f-470c-9af9-54f119f58611","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:38.606548807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 508 len 12288 data: 12 12 11 02/30 Write block 581 len 4096 data: 6 03/30 Write block 368 len 40960 data: 4 3 3 4 4 7 7 8 6 8 04/30 Write block 170 len 40960 data: 8 9 7 5 6 6 6 7 7 6 05/30 Flush 06/30 Write block 311 len 20480 data: 7 8 9 9 9 07/30 Read block 192 len 4096 08/30 Write block 231 len 8192 data: 7 6 09/30 Write block 374 len 36864 data: 8 9 7 9 12 14 14 16 14 10/30 Write block 443 len 16384 data: 9 8 10 10 11/30 Write block 461 len 32768 data: 7 7 7 7 9 7 6 7 12/30 Flush 13/30 Write block 155 len 36864 data: 5 6 6 5 5 6 7 10 10 14/30 Flush 15/30 Write block 384 len 12288 data: 14 11 11 16/30 Write block 418 len 28672 data: 5 5 7 8 8 8 9 17/30 Flush 18/30 Write block 579 len 36864 data: 8 5 7 6 6 6 5 4 5 19/30 Write block 546 len 40960 data: 7 7 8 11 9 8 10 8 7 7 20/30 Read block 189 len 28672 21/30 Write block 68 len 8192 data: 12 9 22/30 Write block 246 len 36864 data: 8 8 9 6 6 5 4 6 6 23/30 Read block 236 len 24576 24/30 Read block 56 len 32768 25/30 Write block 156 len 8192 data: 7 7 26/30 Write block 376 len 32768 data: 8 10 13 15 15 17 15 13 27/30 Read block 319 len 32768 28/30 Write block 397 len 20480 data: 5 7 11 11 13 29/30 Write block 100 len 40960 data: 5 5 5 5 6 6 5 3 5 6 30/30 Read block 460 len 24576 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Write 2 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 10 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 30 2 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:38.634375319Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1568} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 170 170 170 F F F 1 020-039 93 93 93 170 170 170 F F F 2 040-059 88 88 88 163 163 163 F F F 3 060-079 90 90 90 165 165 165 T F T <--- 4 080-099 90 90 90 165 165 165 F F F 5 100-119 88 88 88 163 163 163 T F T <--- 6 120-139 90 90 90 165 165 165 F F F 7 140-159 94 85 94 173 155 173 T F T <--- 8 160-179 94 92 94 173 168 173 F F F <--- 9 180-199 92 92 92 168 168 168 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 94 85 94 172 155 172 F F F <--- 12 240-259 93 93 93 170 170 170 T F T <--- 13 260-279 93 93 93 170 170 170 F F F 14 280-299 91 91 91 166 166 166 F F F 15 300-319 94 88 94 172 163 172 F F F <--- 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 91 91 166 166 166 F F F 18 360-379 94 88 94 172 163 172 T F T <--- 19 380-399 94 88 94 174 163 174 T F T <--- 20 400-419 94 93 94 174 170 174 T F T <--- 21 420-439 94 91 94 174 166 174 F F F <--- 22 440-459 94 93 94 172 170 172 F F F <--- 23 460-479 94 88 94 172 160 172 F F F <--- 24 480-499 91 91 91 166 166 166 F F F 25 500-519 94 93 94 171 170 171 F T F <--- 26 520-539 90 90 90 165 165 165 F F F 27 540-559 93 93 93 170 170 170 T F T <--- 28 560-579 92 92 92 168 168 168 T F T <--- 29 580-599 94 78 94 171 140 171 T T T <--- Max gen: 94, Max flush: 174 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 170 170 170 F F F 1 020-039 93 93 93 170 170 170 F F F 2 040-059 88 88 88 163 163 163 F F F 3 060-079 95 95 95 175 175 175 F F F 4 080-099 90 90 90 165 165 165 F F F 5 100-119 95 95 95 175 175 175 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 95 95 95 175 175 175 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 92 92 92 168 168 168 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 94 94 94 172 172 172 F F F 12 240-259 95 95 95 175 175 175 F F F 13 260-279 93 93 93 170 170 170 F F F 14 280-299 91 91 91 166 166 166 F F F 15 300-319 94 94 94 172 172 172 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 91 91 166 166 166 F F F 18 360-379 95 95 95 175 175 175 F F F 19 380-399 95 95 95 175 175 175 F F F 20 400-419 95 95 95 175 175 175 F F F 21 420-439 94 94 94 174 174 174 F F F 22 440-459 94 94 94 172 172 172 F F F 23 460-479 94 94 94 172 172 172 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 94 94 94 171 171 171 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 95 95 95 175 175 175 F F F 28 560-579 95 95 95 175 175 175 F F F 29 580-599 95 95 95 175 175 175 F F F Max gen: 95, Max flush: 175 Begin loop 48 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.189318593Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:43.210035165Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.210099622Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.225910803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.226383676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.226460608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.226484054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.226518332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"Crucible cc82d88e-bb2e-4f7a-aded-ec43d7168bc6 has session id: d73ae0ca-4571-44ed-a657-052cc4847f18","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.227681227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Upstairs opts: Upstairs UUID: cc82d88e-bb2e-4f7a-aded-ec43d7168bc6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.22774891Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Crucible stats registered with UUID: cc82d88e-bb2e-4f7a-aded-ec43d7168bc6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.227777877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:43.227807696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228195361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228246913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228282654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228608587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228641592Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"cc82d88e-bb2e-4f7a-aded-ec43d7168bc6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.228910943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229009428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229051992Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229278288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229354148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229393145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229516899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"ds_connection connected from Ok(127.0.0.1:62308)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229551057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"ds_connection connected from Ok(127.0.0.1:49561)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.229920776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"ds_connection connected from Ok(127.0.0.1:37285)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.230038378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.231302082Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.231464743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.231978817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232073444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232232989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[0]R flush_numbers[0..12]: [170, 170, 163, 175, 165, 175, 165, 175, 173, 168, 157, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232787984Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[0]R generation[0..12]: [93, 93, 88, 95, 90, 95, 90, 95, 94, 92, 86, 94]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232822272Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232853694Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[1]R flush_numbers[0..12]: [170, 170, 163, 175, 165, 175, 165, 175, 173, 168, 157, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232883022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[1]R generation[0..12]: [93, 93, 88, 95, 90, 95, 90, 95, 94, 92, 86, 94]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232911799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.2329427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[2]R flush_numbers[0..12]: [170, 170, 163, 175, 165, 175, 165, 175, 173, 168, 157, 172]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.232972318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[2]R generation[0..12]: [93, 93, 88, 95, 90, 95, 90, 95, 94, 92, 86, 94]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233001235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233032296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233057045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233090451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"Wait for a query_work_queue command to finish before sending IO msg":"Next flush: 176","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233125961Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233152403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"":"downstairs","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233178244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233201861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"0","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233228173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"1","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233253823Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"client":"2","":"downstairs","session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233279905Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233304413Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"cc82d88e-bb2e-4f7a-aded-ec43d7168bc6 is now active with session: d73ae0ca-4571-44ed-a657-052cc4847f18","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233332178Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233355444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576,"session_id":"d73ae0ca-4571-44ed-a657-052cc4847f18"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233379872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} {"msg":"Activated sub_volume cc82d88e-bb2e-4f7a-aded-ec43d7168bc6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:43.233401184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 394 len 4096 data: 5 03/30 Write block 311 len 12288 data: 8 9 10 04/30 Read block 478 len 20480 05/30 Write block 94 len 4096 data: 9 06/30 Read block 543 len 24576 07/30 Read block 397 len 24576 08/30 Flush 09/30 Flush 10/30 Write block 344 len 4096 data: 5 11/30 Read block 368 len 36864 12/30 Write block 90 len 36864 data: 13 11 12 11 10 9 5 4 4 13/30 Write block 455 len 16384 data: 8 7 10 11 14/30 Write block 83 len 16384 data: 11 10 12 10 15/30 Read block 276 len 16384 16/30 Flush 17/30 Write block 14 len 20480 data: 9 9 7 7 5 18/30 Write block 383 len 32768 data: 14 15 12 12 10 10 6 8 19/30 Write block 189 len 8192 data: 7 6 20/30 Write block 278 len 36864 data: 14 12 12 13 13 11 11 11 11 21/30 Read block 34 len 4096 22/30 Flush 23/30 Write block 232 len 36864 data: 7 4 7 7 8 8 7 9 10 24/30 Write block 7 len 16384 data: 3 5 6 9 25/30 Flush 26/30 Write block 57 len 8192 data: 7 6 27/30 Write block 143 len 4096 data: 6 28/30 Read block 212 len 20480 29/30 Read block 87 len 16384 30/30 Write block 76 len 32768 data: 13 13 11 10 10 12 14 12 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 1 Done Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:43.25731766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1576} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 96 96 170 181 181 F F F <--- 1 020-039 93 93 93 170 170 170 F F F 2 040-059 88 88 88 163 163 163 F T T <--- 3 060-079 95 95 95 175 175 175 F T T <--- 4 080-099 90 96 96 165 179 179 F T T <--- 5 100-119 95 95 95 175 175 175 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 95 95 95 175 175 175 F T T <--- 8 160-179 94 94 94 173 173 173 F F F 9 180-199 92 96 96 168 180 180 F F F <--- 10 200-219 86 86 86 157 157 157 F F F 11 220-239 94 96 96 172 181 181 F F F <--- 12 240-259 95 96 96 175 181 181 F F F <--- 13 260-279 93 96 96 170 180 180 F F F <--- 14 280-299 91 96 96 166 180 180 F F F <--- 15 300-319 94 96 96 172 177 177 F F F <--- 16 320-339 75 75 75 133 133 133 F F F 17 340-359 91 96 96 166 179 179 F F F <--- 18 360-379 95 95 95 175 175 175 F F F 19 380-399 95 96 96 175 180 180 T F F <--- 20 400-419 95 95 95 175 175 175 F F F 21 420-439 94 94 94 174 174 174 F F F 22 440-459 94 96 96 172 179 179 F F F <--- 23 460-479 94 94 94 172 172 172 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 94 94 94 171 171 171 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 95 95 95 175 175 175 F F F 28 560-579 95 95 95 175 175 175 F F F 29 580-599 95 95 95 175 175 175 F F F Max gen: 96, Max flush: 181 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 96 96 96 181 181 181 F F F 1 020-039 93 93 93 170 170 170 F F F 2 040-059 97 97 97 182 182 182 F F F 3 060-079 97 97 97 182 182 182 F F F 4 080-099 97 97 97 182 182 182 F F F 5 100-119 95 95 95 175 175 175 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 96 96 96 180 180 180 F F F 10 200-219 86 86 86 157 157 157 F F F 11 220-239 96 96 96 181 181 181 F F F 12 240-259 96 96 96 181 181 181 F F F 13 260-279 96 96 96 180 180 180 F F F 14 280-299 96 96 96 180 180 180 F F F 15 300-319 96 96 96 177 177 177 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 96 96 96 179 179 179 F F F 18 360-379 95 95 95 175 175 175 F F F 19 380-399 96 96 96 180 180 180 F F F 20 400-419 95 95 95 175 175 175 F F F 21 420-439 94 94 94 174 174 174 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 94 94 94 172 172 172 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 94 94 94 171 171 171 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 95 95 95 175 175 175 F F F 28 560-579 95 95 95 175 175 175 F F F 29 580-599 95 95 95 175 175 175 F F F Max gen: 97, Max flush: 182 Begin loop 49 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.815779066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.838911204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.839452282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.839572008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.839634131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.839682145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"Crucible 93806495-7e1c-4810-911e-0f2797325921 has session id: cb5d2ac6-353d-4418-9087-8e2341d0c646","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.840352198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Upstairs opts: Upstairs UUID: 93806495-7e1c-4810-911e-0f2797325921, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.84046488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Crucible stats registered with UUID: 93806495-7e1c-4810-911e-0f2797325921","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.840518877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:47.840569116Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.840900079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.840981279Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.84103223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841505403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841563588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"93806495-7e1c-4810-911e-0f2797325921 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841730979Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841785145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841834783Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841875614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.84190428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841933368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.841961944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"ds_connection connected from Ok(127.0.0.1:40898)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.842454176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"ds_connection connected from Ok(127.0.0.1:52127)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.842684109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"ds_connection connected from Ok(127.0.0.1:57247)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.842728858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.844568077Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.844668084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.844718524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.845777654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.845834176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[0]R flush_numbers[0..12]: [181, 170, 182, 182, 182, 175, 165, 182, 173, 180, 157, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.845971727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[0]R generation[0..12]: [96, 93, 97, 97, 97, 95, 90, 97, 94, 96, 86, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846014572Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846052476Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[1]R flush_numbers[0..12]: [181, 170, 182, 182, 182, 175, 165, 182, 173, 180, 157, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846097946Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[1]R generation[0..12]: [96, 93, 97, 97, 97, 95, 90, 97, 94, 96, 86, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846144318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846198134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[2]R flush_numbers[0..12]: [181, 170, 182, 182, 182, 175, 165, 182, 173, 180, 157, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846249295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[2]R generation[0..12]: [96, 93, 97, 97, 97, 95, 90, 97, 94, 96, 86, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846284124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846321597Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846351466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846382518Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Next flush: 183","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846412006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846441143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"":"downstairs","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846472135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846500541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"0","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846531422Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"1","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846568435Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"client":"2","":"downstairs","session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846615828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846666167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"93806495-7e1c-4810-911e-0f2797325921 is now active with session: cb5d2ac6-353d-4418-9087-8e2341d0c646","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:03:47.846703942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846742127Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584,"session_id":"cb5d2ac6-353d-4418-9087-8e2341d0c646"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846771545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} {"msg":"Activated sub_volume 93806495-7e1c-4810-911e-0f2797325921","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:47.846797026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 547 len 20480 data: 8 9 12 10 9 02/30 Read block 591 len 36864 03/30 Write block 209 len 36864 data: 4 5 3 5 8 7 4 3 3 04/30 Write block 500 len 16384 data: 9 11 11 10 05/30 Read block 522 len 40960 06/30 Flush 07/30 Write block 192 len 4096 data: 6 08/30 Read block 149 len 16384 09/30 Write block 349 len 12288 data: 6 7 6 10/30 Read block 395 len 8192 11/30 Write block 5 len 4096 data: 4 12/30 Read block 285 len 24576 13/30 Write block 294 len 16384 data: 7 10 10 11 14/30 Read block 123 len 16384 15/30 Write block 368 len 40960 data: 5 4 4 5 5 8 9 10 9 11 16/30 Read block 472 len 36864 17/30 Write block 45 len 8192 data: 8 9 18/30 Write block 365 len 4096 data: 8 19/30 Read block 27 len 40960 20/30 Read block 408 len 28672 21/30 Write block 13 len 36864 data: 12 10 10 8 8 6 6 6 7 22/30 Flush 23/30 Write block 427 len 4096 data: 6 24/30 Read block 166 len 12288 25/30 Write block 570 len 8192 data: 12 9 26/30 Write block 470 len 24576 data: 6 8 7 8 8 9 27/30 Write block 282 len 12288 data: 14 12 12 28/30 Write block 109 len 8192 data: 7 9 29/30 Read block 403 len 32768 30/30 Write block 558 len 28672 data: 6 6 3 3 5 6 7 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 9 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 2 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:47.875658961Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1584} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 96 98 184 181 184 F F F <--- 1 020-039 98 93 98 184 170 184 F F F <--- 2 040-059 98 97 98 184 182 184 F F F <--- 3 060-079 97 97 97 182 182 182 F F F 4 080-099 97 97 97 182 182 182 F F F 5 100-119 95 95 95 175 175 175 T F T <--- 6 120-139 90 90 90 165 165 165 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 98 96 98 184 180 184 F F F <--- 10 200-219 98 86 98 183 157 183 F F F <--- 11 220-239 96 96 96 181 181 181 F F F 12 240-259 96 96 96 181 181 181 F F F 13 260-279 96 96 96 180 180 180 F F F 14 280-299 98 96 98 184 180 184 T F T <--- 15 300-319 96 96 96 177 177 177 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 98 96 98 184 179 184 F F F <--- 18 360-379 98 95 98 184 175 184 F F F <--- 19 380-399 96 96 96 180 180 180 F F F 20 400-419 95 95 95 175 175 175 F F F 21 420-439 94 94 94 174 174 174 T F T <--- 22 440-459 96 96 96 179 179 179 F F F 23 460-479 94 94 94 172 172 172 T F T <--- 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 94 98 183 171 183 F F F <--- 26 520-539 90 90 90 165 165 165 F F F 27 540-559 98 95 98 183 175 183 T T T <--- 28 560-579 95 95 95 175 175 175 T F T <--- 29 580-599 95 95 95 175 175 175 F F F Max gen: 98, Max flush: 184 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 184 184 184 F F F 1 020-039 98 98 98 184 184 184 F F F 2 040-059 98 98 98 184 184 184 F F F 3 060-079 97 97 97 182 182 182 F F F 4 080-099 97 97 97 182 182 182 F F F 5 100-119 99 99 99 185 185 185 F F F 6 120-139 90 90 90 165 165 165 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 98 98 98 184 184 184 F F F 10 200-219 98 98 98 183 183 183 F F F 11 220-239 96 96 96 181 181 181 F F F 12 240-259 96 96 96 181 181 181 F F F 13 260-279 96 96 96 180 180 180 F F F 14 280-299 99 99 99 185 185 185 F F F 15 300-319 96 96 96 177 177 177 F F F 16 320-339 75 75 75 133 133 133 F F F 17 340-359 98 98 98 184 184 184 F F F 18 360-379 98 98 98 184 184 184 F F F 19 380-399 96 96 96 180 180 180 F F F 20 400-419 95 95 95 175 175 175 F F F 21 420-439 99 99 99 185 185 185 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 99 99 99 185 185 185 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 99 99 99 185 185 185 F F F 29 580-599 95 95 95 175 175 175 F F F Max gen: 99, Max flush: 185 Begin loop 50 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.441680016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.46354825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.464244945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.464327157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.464352577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.464390041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"Crucible e1b80283-e08b-4e95-ba2c-33b9acc71580 has session id: 2427ef95-31c3-49d4-975b-d24b80e48f91","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.465625379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Upstairs opts: Upstairs UUID: e1b80283-e08b-4e95-ba2c-33b9acc71580, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.465744805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Crucible stats registered with UUID: e1b80283-e08b-4e95-ba2c-33b9acc71580","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.465789102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:52.465817207Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466188711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466258478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466288878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466792803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466835216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"e1b80283-e08b-4e95-ba2c-33b9acc71580 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.466944722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.4669787Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467007556Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467046042Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467086382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467129788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467700794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"ds_connection connected from Ok(127.0.0.1:35145)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467755663Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"ds_connection connected from Ok(127.0.0.1:53117)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.467986037Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"ds_connection connected from Ok(127.0.0.1:38640)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.468463269Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.469296012Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.469386501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.46974587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470362277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470518745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[0]R flush_numbers[0..12]: [184, 184, 184, 182, 182, 185, 165, 182, 173, 184, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470700734Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[0]R generation[0..12]: [98, 98, 98, 97, 97, 99, 90, 97, 94, 98, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470737287Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470770833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[1]R flush_numbers[0..12]: [184, 184, 184, 182, 182, 185, 165, 182, 173, 184, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470803137Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[1]R generation[0..12]: [98, 98, 98, 97, 97, 99, 90, 97, 94, 98, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470834027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470866942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[2]R flush_numbers[0..12]: [184, 184, 184, 182, 182, 185, 165, 182, 173, 184, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470898555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[2]R generation[0..12]: [98, 98, 98, 97, 97, 99, 90, 97, 94, 98, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470929145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.47096208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.470989164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471023321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Next flush: 186","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471050655Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471077278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"":"downstairs","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471105463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.47113928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"0","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471168357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"1","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471196322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"client":"2","":"downstairs","session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471224688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471251451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"e1b80283-e08b-4e95-ba2c-33b9acc71580 is now active with session: 2427ef95-31c3-49d4-975b-d24b80e48f91","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471281491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.47130674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592,"session_id":"2427ef95-31c3-49d4-975b-d24b80e48f91"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471333523Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Activated sub_volume e1b80283-e08b-4e95-ba2c-33b9acc71580","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:52.471362991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 333 len 36864 data: 8 7 7 8 7 7 6 4 6 02/30 Read block 435 len 20480 03/30 Write block 122 len 36864 data: 7 7 8 7 7 8 6 4 9 04/30 Read block 36 len 28672 05/30 Read block 430 len 12288 06/30 Flush 07/30 Write block 334 len 40960 data: 8 8 9 8 8 7 5 7 6 5 08/30 Write block 435 len 32768 data: 9 10 10 10 8 7 10 9 09/30 Write block 377 len 12288 data: 12 14 16 10/30 Write block 321 len 12288 data: 6 4 3 11/30 Read block 236 len 32768 12/30 Read block 48 len 8192 13/30 Flush 14/30 Write block 184 len 12288 data: 9 10 10 15/30 Write block 318 len 28672 data: 7 6 7 7 5 4 3 16/30 Write block 346 len 16384 data: 8 9 8 7 17/30 Read block 265 len 32768 18/30 Write block 103 len 40960 data: 6 7 7 6 4 6 8 10 9 12 19/30 Read block 22 len 28672 20/30 Write block 101 len 28672 data: 6 6 7 8 8 7 5 21/30 Read block 179 len 8192 22/30 Read block 77 len 28672 23/30 Read block 280 len 8192 24/30 Read block 47 len 20480 25/30 Read block 317 len 4096 26/30 Write block 571 len 40960 data: 10 8 8 8 7 7 7 6 9 6 27/30 Write block 409 len 16384 data: 5 6 7 8 28/30 Read block 361 len 8192 29/30 Write block 74 len 8192 data: 11 13 30/30 Read block 377 len 24576 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Sent false 1001 Acked Read 5 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 30 31 Done 29 30 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1012)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:52.503602271Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1592} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 184 184 184 F F F 1 020-039 98 98 98 184 184 184 F F F 2 040-059 98 98 98 184 184 184 F F F 3 060-079 97 97 97 182 182 182 T T F <--- 4 080-099 97 97 97 182 182 182 F F F 5 100-119 99 99 99 185 185 185 T T F <--- 6 120-139 100 100 90 186 186 165 F F F <--- 7 140-159 97 97 97 182 182 182 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 98 98 98 184 184 184 T T F <--- 10 200-219 98 98 98 183 183 183 F F F 11 220-239 96 96 96 181 181 181 F F F 12 240-259 96 96 96 181 181 181 F F F 13 260-279 96 96 96 180 180 180 F F F 14 280-299 99 99 99 185 185 185 F F F 15 300-319 96 96 96 177 177 177 T T F <--- 16 320-339 100 100 75 187 187 133 T T F <--- 17 340-359 100 100 98 187 187 184 T T F <--- 18 360-379 100 100 98 187 187 184 F F F <--- 19 380-399 96 96 96 180 180 180 F F F 20 400-419 95 95 95 175 175 175 T T F <--- 21 420-439 100 100 99 187 187 185 F F F <--- 22 440-459 100 100 96 187 187 179 F F F <--- 23 460-479 99 99 99 185 185 185 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 99 99 99 185 185 185 T T F <--- 29 580-599 95 95 95 175 175 175 T T F <--- Max gen: 100, Max flush: 187 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 184 184 184 F F F 1 020-039 98 98 98 184 184 184 F F F 2 040-059 98 98 98 184 184 184 F F F 3 060-079 101 101 101 188 188 188 F F F 4 080-099 97 97 97 182 182 182 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 94 94 94 173 173 173 F F F 9 180-199 101 101 101 188 188 188 F F F 10 200-219 98 98 98 183 183 183 F F F 11 220-239 96 96 96 181 181 181 F F F 12 240-259 96 96 96 181 181 181 F F F 13 260-279 96 96 96 180 180 180 F F F 14 280-299 99 99 99 185 185 185 F F F 15 300-319 101 101 101 188 188 188 F F F 16 320-339 101 101 101 188 188 188 F F F 17 340-359 101 101 101 188 188 188 F F F 18 360-379 100 100 100 187 187 187 F F F 19 380-399 96 96 96 180 180 180 F F F 20 400-419 101 101 101 188 188 188 F F F 21 420-439 100 100 100 187 187 187 F F F 22 440-459 100 100 100 187 187 187 F F F 23 460-479 99 99 99 185 185 185 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 101, Max flush: 188 Begin loop 51 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.084328814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.106373055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.10688741Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.106965675Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.107027407Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.107066915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"Crucible b95ebda3-21d0-48c7-b002-13510b4f39f7 has session id: 9bf92029-947d-49ff-ba4c-ca5a38ef288a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108129712Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Upstairs opts: Upstairs UUID: b95ebda3-21d0-48c7-b002-13510b4f39f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108197345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Crucible stats registered with UUID: b95ebda3-21d0-48c7-b002-13510b4f39f7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108227475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:57.108253827Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108558538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108617214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.108647183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109081099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109115748Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"b95ebda3-21d0-48c7-b002-13510b4f39f7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109389257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109421742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109450629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109485968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109514324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109543472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109571818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"ds_connection connected from Ok(127.0.0.1:63443)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109736843Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"ds_connection connected from Ok(127.0.0.1:60839)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.109772023Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"ds_connection connected from Ok(127.0.0.1:63324)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.110030924Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.112206016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.112363457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.11260331Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.11277049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113101984Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[0]R flush_numbers[0..12]: [184, 184, 184, 188, 182, 188, 186, 182, 173, 188, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113543385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[0]R generation[0..12]: [98, 98, 98, 101, 97, 101, 100, 97, 94, 101, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113629836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113665105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[1]R flush_numbers[0..12]: [184, 184, 184, 188, 182, 188, 186, 182, 173, 188, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113705615Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[1]R generation[0..12]: [98, 98, 98, 101, 97, 101, 100, 97, 94, 101, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113737719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113770774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[2]R flush_numbers[0..12]: [184, 184, 184, 188, 182, 188, 186, 182, 173, 188, 183, 181]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113802516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[2]R generation[0..12]: [98, 98, 98, 101, 97, 101, 100, 97, 94, 101, 98, 96]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113833427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113866402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113893817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113922243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Next flush: 189","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113949486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.113976209Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"":"downstairs","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114004816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114039163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"0","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114068932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"1","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114097018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"client":"2","":"downstairs","session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114125704Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114155593Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"b95ebda3-21d0-48c7-b002-13510b4f39f7 is now active with session: 9bf92029-947d-49ff-ba4c-ca5a38ef288a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114185773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114210942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600,"session_id":"9bf92029-947d-49ff-ba4c-ca5a38ef288a"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114237675Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} {"msg":"Activated sub_volume b95ebda3-21d0-48c7-b002-13510b4f39f7","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:03:57.114261011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 328 len 12288 data: 6 7 8 02/30 Write block 283 len 12288 data: 13 13 12 03/30 Read block 62 len 12288 04/30 Write block 259 len 8192 data: 12 10 05/30 Write block 219 len 28672 data: 4 3 3 4 4 5 6 06/30 Read block 511 len 20480 07/30 Write block 341 len 8192 data: 8 7 08/30 Read block 576 len 20480 09/30 Read block 505 len 32768 10/30 Read block 432 len 32768 11/30 Flush 12/30 Write block 309 len 32768 data: 7 6 9 10 11 10 10 8 13/30 Flush 14/30 Write block 162 len 4096 data: 11 15/30 Read block 194 len 16384 16/30 Read block 407 len 28672 17/30 Write block 437 len 8192 data: 11 11 18/30 Read block 496 len 40960 19/30 Write block 194 len 8192 data: 9 9 20/30 Read block 523 len 32768 21/30 Read block 429 len 36864 22/30 Flush 23/30 Write block 76 len 8192 data: 14 14 24/30 Read block 405 len 16384 25/30 Flush 26/30 Write block 379 len 28672 data: 17 16 18 16 15 16 13 27/30 Read block 23 len 8192 28/30 Read block 5 len 8192 29/30 Write block 82 len 20480 data: 15 13 11 13 11 30/30 Read block 181 len 40960 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 0 29 Done 29 2 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:03:57.145157261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1600} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 184 184 184 F F F 1 020-039 98 98 98 184 184 184 F F F 2 040-059 98 98 98 184 184 184 F F F 3 060-079 102 101 102 192 188 192 F F F <--- 4 080-099 97 97 97 182 182 182 T F T <--- 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 102 94 102 191 173 191 F F F <--- 9 180-199 102 101 102 191 188 191 F F F <--- 10 200-219 102 98 102 189 183 189 F F F <--- 11 220-239 102 96 102 189 181 189 F F F <--- 12 240-259 102 96 102 189 181 189 F F F <--- 13 260-279 102 96 102 189 180 189 F F F <--- 14 280-299 102 99 102 189 185 189 F T F <--- 15 300-319 102 101 102 190 188 190 F F F <--- 16 320-339 102 101 102 189 188 189 F T F <--- 17 340-359 102 101 102 189 188 189 F F F <--- 18 360-379 100 100 100 187 187 187 T F T <--- 19 380-399 96 96 96 180 180 180 T F T <--- 20 400-419 101 101 101 188 188 188 F F F 21 420-439 102 100 102 191 187 191 F F F <--- 22 440-459 100 100 100 187 187 187 F F F 23 460-479 99 99 99 185 185 185 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 102, Max flush: 192 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 184 184 184 F F F 1 020-039 98 98 98 184 184 184 F F F 2 040-059 98 98 98 184 184 184 F F F 3 060-079 102 102 102 192 192 192 F F F 4 080-099 103 103 103 193 193 193 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 102 102 102 191 191 191 F F F 9 180-199 102 102 102 191 191 191 F F F 10 200-219 102 102 102 189 189 189 F F F 11 220-239 102 102 102 189 189 189 F F F 12 240-259 102 102 102 189 189 189 F F F 13 260-279 102 102 102 189 189 189 F F F 14 280-299 102 102 102 189 189 189 F F F 15 300-319 102 102 102 190 190 190 F F F 16 320-339 102 102 102 189 189 189 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 103 103 103 193 193 193 F F F 20 400-419 101 101 101 188 188 188 F F F 21 420-439 102 102 102 191 191 191 F F F 22 440-459 100 100 100 187 187 187 F F F 23 460-479 99 99 99 185 185 185 F F F 24 480-499 91 91 91 166 166 166 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 103, Max flush: 193 Begin loop 52 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.718980242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.741010776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.741569329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.741649898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.741675318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.741712291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"Crucible 33068682-d914-4680-a1fd-329e262b1ed6 has session id: 9d31857a-b137-4a58-8e75-66a9f66b6b61","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.742533371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Upstairs opts: Upstairs UUID: 33068682-d914-4680-a1fd-329e262b1ed6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.742598079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Crucible stats registered with UUID: 33068682-d914-4680-a1fd-329e262b1ed6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.74262888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:01.742655442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743249564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743305365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743335073Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743588062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743621689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"33068682-d914-4680-a1fd-329e262b1ed6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743726566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743756625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743785101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743813387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743841222Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.743870079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"ds_connection connected from Ok(127.0.0.1:57059)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.744251872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.7443417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"ds_connection connected from Ok(127.0.0.1:51842)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.744374645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"ds_connection connected from Ok(127.0.0.1:64629)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.744405506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.746037376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.746100801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.746134908Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.746829619Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.746873486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[0]R flush_numbers[0..12]: [184, 184, 184, 192, 193, 188, 186, 182, 191, 191, 189, 189]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747031337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[0]R generation[0..12]: [98, 98, 98, 102, 103, 101, 100, 97, 102, 102, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747069633Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747103359Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[1]R flush_numbers[0..12]: [184, 184, 184, 192, 193, 188, 186, 182, 191, 191, 189, 189]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747136555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[1]R generation[0..12]: [98, 98, 98, 102, 103, 101, 100, 97, 102, 102, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747167927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747200822Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[2]R flush_numbers[0..12]: [184, 184, 184, 192, 193, 188, 186, 182, 191, 191, 189, 189]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747232474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[2]R generation[0..12]: [98, 98, 98, 102, 103, 101, 100, 97, 102, 102, 102, 102]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747263996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747297112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747324105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747358453Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Next flush: 194","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747385616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747412359Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"":"downstairs","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747440394Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747466366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"0","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747494281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"1","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747521975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"client":"2","":"downstairs","session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747550371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"Wait for a query_work_queue command to finish before sending IO 9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747585962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"33068682-d914-4680-a1fd-329e262b1ed6 is now active with session: 9d31857a-b137-4a58-8e75-66a9f66b6b61","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747616402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747641642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608,"session_id":"9d31857a-b137-4a58-8e75-66a9f66b6b61"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.747668324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} {"msg":"Activated sub_volume 33068682-d914-4680-a1fd-329e262b1ed6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:01.74769152Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 322 len 36864 data: 6 5 4 5 5 6 7 8 9 02/30 Read block 433 len 4096 03/30 Read block 583 len 12288 04/30 Write block 6 len 20480 data: 5 4 6 7 10 05/30 Read block 503 len 8192 06/30 Read block 572 len 24576 07/30 Write block 491 len 16384 data: 5 5 6 6 08/30 Read block 307 len 28672 09/30 Write block 470 len 40960 data: 7 9 8 9 9 10 10 10 11 9 10/30 Write block 408 len 36864 data: 7 6 7 8 9 7 9 7 7 11/30 Read block 523 len 12288 12/30 Write block 37 len 24576 data: 11 11 8 6 4 4 13/30 Read block 335 len 4096 14/30 Write block 211 len 16384 data: 4 6 9 8 15/30 Write block 420 len 32768 data: 8 9 9 9 10 6 4 7 16/30 Read block 342 len 4096 17/30 Read block 369 len 12288 18/30 Read block 29 len 36864 19/30 Write block 8 len 28672 data: 7 8 11 11 12 13 11 20/30 Flush 21/30 Write block 214 len 28672 data: 9 5 4 4 4 5 4 22/30 Read block 203 len 28672 23/30 Read block 33 len 20480 24/30 Flush 25/30 Write block 385 len 4096 data: 14 26/30 Read block 564 len 28672 27/30 Read block 111 len 20480 28/30 Write block 168 len 16384 data: 6 7 9 10 29/30 Write block 2 len 12288 data: 4 4 4 30/30 Read block 310 len 8192 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Write 4 Done Done Done false 1007 Acked Read 7 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 23 23 Done 30 30 7 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:01.78073643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1608} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 98 194 194 184 T T T <--- 1 020-039 104 104 98 194 194 184 F F F <--- 2 040-059 104 104 98 194 194 184 F F F <--- 3 060-079 102 102 102 192 192 192 F F F 4 080-099 103 103 103 193 193 193 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 102 102 102 191 191 191 T T F <--- 9 180-199 102 102 102 191 191 191 F F F 10 200-219 104 104 102 195 195 189 F F F <--- 11 220-239 104 104 102 195 195 189 F F F <--- 12 240-259 102 102 102 189 189 189 F F F 13 260-279 102 102 102 189 189 189 F F F 14 280-299 102 102 102 189 189 189 F F F 15 300-319 102 102 102 190 190 190 F F F 16 320-339 104 104 102 194 194 189 F F T <--- 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 103 103 103 193 193 193 T T F <--- 20 400-419 104 104 101 194 194 188 F F F <--- 21 420-439 104 104 102 194 194 191 F F F <--- 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 99 194 194 185 F F F <--- 24 480-499 104 104 91 194 194 166 F F T <--- 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 104, Max flush: 195 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 105 105 105 196 196 196 F F F 1 020-039 104 104 104 194 194 194 F F F 2 040-059 104 104 104 194 194 194 F F F 3 060-079 102 102 102 192 192 192 F F F 4 080-099 103 103 103 193 193 193 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 105 105 105 196 196 196 F F F 9 180-199 102 102 102 191 191 191 F F F 10 200-219 104 104 104 195 195 195 F F F 11 220-239 104 104 104 195 195 195 F F F 12 240-259 102 102 102 189 189 189 F F F 13 260-279 102 102 102 189 189 189 F F F 14 280-299 102 102 102 189 189 189 F F F 15 300-319 102 102 102 190 190 190 F F F 16 320-339 104 104 104 194 194 194 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 105 105 105 196 196 196 F F F 20 400-419 104 104 104 194 194 194 F F F 21 420-439 104 104 104 194 194 194 F F F 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 104 194 194 194 F F F 24 480-499 104 104 104 194 194 194 F F F 25 500-519 98 98 98 183 183 183 F F F 26 520-539 90 90 90 165 165 165 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 105, Max flush: 196 Begin loop 53 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.322737204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.343512872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.344048659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.344161342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.344193745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.344231199Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"Crucible 9e387ef0-c8c6-4033-a97c-2d4ec0e892c4 has session id: 35ac53af-07e3-46dd-a8cd-5aea5dee2833","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345373564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Upstairs opts: Upstairs UUID: 9e387ef0-c8c6-4033-a97c-2d4ec0e892c4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345436097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Crucible stats registered with UUID: 9e387ef0-c8c6-4033-a97c-2d4ec0e892c4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345464814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:06.345489392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345788212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345847719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.345875194Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346051321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346083665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"9e387ef0-c8c6-4033-a97c-2d4ec0e892c4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346157691Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346185265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346212299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346245184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346271636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346384058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346469286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"ds_connection connected from Ok(127.0.0.1:64990)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346499235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"ds_connection connected from Ok(127.0.0.1:43711)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.346792243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:06.347224927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:06.347320766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:06.347356717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.347513907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.347548756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.348113771Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.34823974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.348843411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:06.349139235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.348722298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"ds_connection connected from Ok(127.0.0.1:48390)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.348842455Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.351707769Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"1","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[0]R flush_numbers[0..12]: [196, 194, 194, 192, 193, 188, 186, 182, 196, 191, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352590051Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[0]R generation[0..12]: [105, 104, 104, 102, 103, 101, 100, 97, 105, 102, 104, 104]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352657634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352705388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[1]R flush_numbers[0..12]: [196, 194, 194, 192, 193, 188, 186, 182, 196, 191, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352750227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[1]R generation[0..12]: [105, 104, 104, 102, 103, 101, 100, 97, 105, 102, 104, 104]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352794484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352842168Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[2]R flush_numbers[0..12]: [196, 194, 194, 192, 193, 188, 186, 182, 196, 191, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352879792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[2]R generation[0..12]: [105, 104, 104, 102, 103, 101, 100, 97, 105, 102, 104, 104]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.352921895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353051722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"Wait for a query_work_queue command to finish before sending IO msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353110989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353159765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Next flush: 197","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353215334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353349068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"":"downstairs","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353390831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353428445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"0","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353472833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"1","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353531308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"client":"2","":"downstairs","session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353564003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353589463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"9e387ef0-c8c6-4033-a97c-2d4ec0e892c4 is now active with session: 35ac53af-07e3-46dd-a8cd-5aea5dee2833","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353617177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353640894Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616,"session_id":"35ac53af-07e3-46dd-a8cd-5aea5dee2833"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353665573Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} {"msg":"Activated sub_volume 9e387ef0-c8c6-4033-a97c-2d4ec0e892c4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:16.353687286Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 519 len 8192 data: 10 11 02/30 Read block 214 len 8192 03/30 Write block 23 len 32768 data: 4 5 4 4 7 9 13 10 04/30 Read block 453 len 32768 05/30 Read block 133 len 40960 06/30 Read block 483 len 24576 07/30 Read block 395 len 32768 08/30 Write block 221 len 12288 data: 4 5 5 09/30 Write block 239 len 32768 data: 10 11 11 13 12 10 10 9 10/30 Read block 575 len 16384 11/30 Write block 219 len 16384 data: 6 5 5 6 12/30 Write block 408 len 16384 data: 8 7 8 9 13/30 Read block 530 len 32768 14/30 Read block 102 len 20480 15/30 Write block 32 len 28672 data: 10 10 12 12 9 12 12 16/30 Write block 219 len 32768 data: 7 6 6 7 6 6 7 8 17/30 Read block 515 len 28672 18/30 Write block 80 len 28672 data: 11 13 16 14 12 14 12 19/30 Read block 257 len 16384 20/30 Flush 21/30 Write block 428 len 12288 data: 7 7 8 22/30 Write block 160 len 36864 data: 7 8 12 11 10 9 12 9 7 23/30 Write block 493 len 36864 data: 7 7 6 8 9 10 10 10 12 24/30 Write block 190 len 4096 data: 7 25/30 Write block 187 len 24576 data: 9 9 8 8 7 7 26/30 Read block 423 len 28672 27/30 Write block 243 len 28672 data: 13 11 11 10 9 10 7 28/30 Write block 168 len 28672 data: 8 8 10 11 8 6 7 29/30 Read block 290 len 12288 30/30 Read block 321 len 12288 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:16.385145514Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1616} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 105 105 105 196 196 196 F F F 1 020-039 106 104 106 197 194 197 F F F <--- 2 040-059 104 104 104 194 194 194 F F F 3 060-079 102 102 102 192 192 192 F F F 4 080-099 106 103 106 197 193 197 F F F <--- 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 105 105 105 196 196 196 T F T <--- 9 180-199 102 102 102 191 191 191 T F T <--- 10 200-219 106 104 106 197 195 197 F F F <--- 11 220-239 106 104 106 197 195 197 F F F <--- 12 240-259 106 102 106 197 189 197 T F T <--- 13 260-279 102 102 102 189 189 189 F F F 14 280-299 102 102 102 189 189 189 F F F 15 300-319 102 102 102 190 190 190 F F F 16 320-339 104 104 104 194 194 194 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 105 105 105 196 196 196 F F F 20 400-419 106 104 106 197 194 197 F F F <--- 21 420-439 104 104 104 194 194 194 T F T <--- 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 104 194 194 194 F F F 24 480-499 104 104 104 194 194 194 T F T <--- 25 500-519 106 98 106 197 183 197 T F T <--- 26 520-539 106 90 106 197 165 197 F F F <--- 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 106, Max flush: 197 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 105 105 105 196 196 196 F F F 1 020-039 106 106 106 197 197 197 F F F 2 040-059 104 104 104 194 194 194 F F F 3 060-079 102 102 102 192 192 192 F F F 4 080-099 106 106 106 197 197 197 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 100 100 100 186 186 186 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 107 107 107 198 198 198 F F F 10 200-219 106 106 106 197 197 197 F F F 11 220-239 106 106 106 197 197 197 F F F 12 240-259 107 107 107 198 198 198 F F F 13 260-279 102 102 102 189 189 189 F F F 14 280-299 102 102 102 189 189 189 F F F 15 300-319 102 102 102 190 190 190 F F F 16 320-339 104 104 104 194 194 194 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 105 105 105 196 196 196 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 107 107 107 198 198 198 F F F 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 104 194 194 194 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 106 106 106 197 197 197 F F F 27 540-559 99 99 99 185 185 185 F F F 28 560-579 101 101 101 188 188 188 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 107, Max flush: 198 Begin loop 54 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.937764824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.961013792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.96153466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.961616001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.961649336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.961689676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"Crucible 279a0c32-1cb3-4834-855b-8def778fd5ff has session id: 6808e9db-553c-4f5a-bd8d-dede9bba6bae","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.962708226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Upstairs opts: Upstairs UUID: 279a0c32-1cb3-4834-855b-8def778fd5ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.962809636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Crucible stats registered with UUID: 279a0c32-1cb3-4834-855b-8def778fd5ff","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.962866689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:20.962913571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.963362517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.96342458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.963454649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.963687238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.963723359Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"279a0c32-1cb3-4834-855b-8def778fd5ff active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.963919947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964180421Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964226813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964251541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964282122Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964308564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964332662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"ds_connection connected from Ok(127.0.0.1:40187)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964357741Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"ds_connection connected from Ok(127.0.0.1:33890)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964487507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"ds_connection connected from Ok(127.0.0.1:41856)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.964541514Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.966084609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.966214625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.966536491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.966822765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.966924456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[0]R flush_numbers[0..12]: [196, 197, 194, 192, 197, 188, 186, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967285649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[0]R generation[0..12]: [105, 106, 104, 102, 106, 101, 100, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967324265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967351659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[1]R flush_numbers[0..12]: [196, 197, 194, 192, 197, 188, 186, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967384634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[1]R generation[0..12]: [105, 106, 104, 102, 106, 101, 100, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967411287Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967438581Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[2]R flush_numbers[0..12]: [196, 197, 194, 192, 197, 188, 186, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967464572Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[2]R generation[0..12]: [105, 106, 104, 102, 106, 101, 100, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967490303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967517526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967539911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967563217Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Next flush: 199","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.96758507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967607013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"":"downstairs","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967629938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.96765114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"0","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967674416Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"1","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967697472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"client":"2","":"downstairs","session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967720798Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967742621Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"279a0c32-1cb3-4834-855b-8def778fd5ff is now active with session: 6808e9db-553c-4f5a-bd8d-dede9bba6bae","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967770876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {Wait for a query_work_queue command to finish before sending IO "msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967819102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624,"session_id":"6808e9db-553c-4f5a-bd8d-dede9bba6bae"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967844131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} {"msg":"Activated sub_volume 279a0c32-1cb3-4834-855b-8def778fd5ff","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:20.967863159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 327 len 20480 data: 7 8 9 10 6 02/30 Write block 127 len 4096 data: 9 03/30 Write block 429 len 24576 data: 8 9 8 5 5 8 04/30 Write block 330 len 8192 data: 11 7 05/30 Read block 481 len 24576 06/30 Read block 462 len 4096 07/30 Read block 520 len 32768 08/30 Read block 115 len 20480 09/30 Write block 243 len 12288 data: 14 12 12 10/30 Read block 41 len 8192 11/30 Read block 492 len 24576 12/30 Write block 556 len 20480 data: 6 6 7 7 4 13/30 Flush 14/30 Write block 67 len 28672 data: 10 13 10 8 6 8 9 15/30 Read block 115 len 12288 16/30 Read block 570 len 12288 17/30 Read block 349 len 24576 18/30 Read block 1 len 24576 19/30 Write block 528 len 16384 data: 10 8 7 8 20/30 Read block 159 len 40960 21/30 Read block 293 len 12288 22/30 Write block 563 len 20480 data: 7 8 10 10 12 23/30 Write block 305 len 32768 data: 6 7 7 8 8 7 10 11 24/30 Read block 235 len 20480 25/30 Flush 26/30 Write block 9 len 40960 data: 9 12 12 13 14 12 11 9 9 7 27/30 Write block 18 len 40960 data: 8 7 7 8 4 5 6 5 5 8 28/30 Write block 285 len 36864 data: 13 12 8 10 13 10 10 9 7 29/30 Write block 278 len 40960 data: 15 13 13 14 15 14 14 14 13 9 30/30 Read block 534 len 12288 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Read 5 Done Done Done false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 10 Done Sent Sent false 1027 Acked Write 9 Done Sent Sent false 1028 Acked Write 10 Done Sent Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 4 22 26 Done 30 26 8 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:20.997375758Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1624} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 105 105 105 196 196 196 T T F <--- 1 020-039 106 106 106 197 197 197 T T F <--- 2 040-059 104 104 104 194 194 194 F F F 3 060-079 108 108 102 200 200 192 F F F <--- 4 080-099 106 106 106 197 197 197 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 108 108 100 199 199 186 F F T <--- 7 140-159 97 97 97 182 182 182 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 107 107 107 198 198 198 F F F 10 200-219 106 106 106 197 197 197 F F F 11 220-239 106 106 106 197 197 197 F F F 12 240-259 108 108 107 199 199 198 F F F <--- 13 260-279 102 102 102 189 189 189 T T F <--- 14 280-299 102 102 102 189 189 189 T T F <--- 15 300-319 108 108 102 200 200 190 F F F <--- 16 320-339 108 108 104 199 199 194 F F T <--- 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 105 105 105 196 196 196 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 107 199 199 198 F F T <--- 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 104 194 194 194 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 108 108 106 200 200 197 F F F <--- 27 540-559 108 108 99 199 199 185 F F F <--- 28 560-579 108 108 101 200 200 188 F F F <--- 29 580-599 101 101 101 188 188 188 F F F Max gen: 108, Max flush: 200 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 201 201 201 F F F 1 020-039 109 109 109 201 201 201 F F F 2 040-059 104 104 104 194 194 194 F F F 3 060-079 108 108 108 200 200 200 F F F 4 080-099 106 106 106 197 197 197 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 108 108 108 199 199 199 F F F 7 140-159 97 97 97 182 182 182 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 107 107 107 198 198 198 F F F 10 200-219 106 106 106 197 197 197 F F F 11 220-239 106 106 106 197 197 197 F F F 12 240-259 108 108 108 199 199 199 F F F 13 260-279 109 109 109 201 201 201 F F F 14 280-299 109 109 109 201 201 201 F F F 15 300-319 108 108 108 200 200 200 F F F 16 320-339 108 108 108 199 199 199 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 F F F 19 380-399 105 105 105 196 196 196 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 100 100 100 187 187 187 F F F 23 460-479 104 104 104 194 194 194 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 108 108 108 200 200 200 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 200 200 200 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 109, Max flush: 201 Begin loop 55 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.541562244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.562112948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.562495563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.562588076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.562643535Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.562683564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"Crucible 5a8098c1-81c9-4009-8d3d-10083e8f55bf has session id: 9f69f890-e313-4530-b22c-d16a83b2b0d3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.563712264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Upstairs opts: Upstairs UUID: 5a8098c1-81c9-4009-8d3d-10083e8f55bf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.564065862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Crucible stats registered with UUID: 5a8098c1-81c9-4009-8d3d-10083e8f55bf","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.564107134Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:25.564139488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.564710855Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.564769441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56479947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565038341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565096045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"5a8098c1-81c9-4009-8d3d-10083e8f55bf active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565196543Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565228006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565256592Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565282423Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56531146Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.565338073Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.566192158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48973)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.566575655Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48104)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56661383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:54342)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.566642046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56748531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.567737818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.567781885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568550442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568632173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[0]R flush_numbers[0..12]: [201, 201, 194, 200, 197, 188, 199, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568664497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[0]R generation[0..12]: [109, 109, 104, 108, 106, 101, 108, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56869666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568728513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[1]R flush_numbers[0..12]: [201, 201, 194, 200, 197, 188, 199, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568758002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[1]R generation[0..12]: [109, 109, 104, 108, 106, 101, 108, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56878753Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56881794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[2]R flush_numbers[0..12]: [201, 201, 194, 200, 197, 188, 199, 182, 198, 198, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568847478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[2]R generation[0..12]: [109, 109, 104, 108, 106, 101, 108, 97, 107, 107, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568876656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568907026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568932115Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568968026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Next flush: 202","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.568992845Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569017734Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"":"downstairs","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569043715Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569067803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"0","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569093954Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"1","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569119575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"client":"2","":"downstairs","session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569145677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569170155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"5a8098c1-81c9-4009-8d3d-10083e8f55bf is now active with session: 9f69f890-e313-4530-b22c-d16a83b2b0d3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.56919788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569221005Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632,"session_id":"9f69f890-e313-4530-b22c-d16a83b2b0d3"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569245564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} {"msg":"Activated sub_volume 5a8098c1-81c9-4009-8d3d-10083e8f55bf","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:25.569267287Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 152 len 24576 data: 7 8 6 6 8 8 02/30 Flush 03/30 Write block 217 len 36864 data: 5 5 8 7 7 8 7 7 8 04/30 Flush 05/30 Write block 3 len 32768 data: 5 5 5 6 5 8 10 13 06/30 Flush 07/30 Write block 81 len 16384 data: 14 17 15 13 08/30 Read block 389 len 32768 09/30 Read block 4 len 4096 10/30 Write block 237 len 12288 data: 9 8 11 11/30 Write block 183 len 20480 data: 13 10 11 11 10 12/30 Write block 538 len 40960 data: 10 12 10 11 12 9 8 8 8 9 13/30 Read block 540 len 24576 14/30 Read block 413 len 28672 15/30 Write block 207 len 24576 data: 7 5 5 6 5 7 16/30 Read block 83 len 12288 17/30 Flush 18/30 Write block 378 len 24576 data: 15 18 17 19 17 16 19/30 Write block 27 len 8192 data: 9 10 20/30 Write block 475 len 4096 data: 11 21/30 Read block 98 len 12288 22/30 Read block 2 len 20480 23/30 Write block 445 len 40960 data: 11 11 9 10 10 9 10 8 9 8 24/30 Read block 515 len 32768 25/30 Read block 1 len 16384 26/30 Read block 137 len 28672 27/30 Write block 576 len 8192 data: 8 8 28/30 Read block 328 len 32768 29/30 Write block 555 len 36864 data: 8 7 7 8 8 5 4 6 8 30/30 Read block 432 len 12288 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 9 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:25.599370441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1632} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 109 204 204 201 F F F <--- 1 020-039 109 109 109 201 201 201 T T F <--- 2 040-059 104 104 104 194 194 194 F F F 3 060-079 108 108 108 200 200 200 F F F 4 080-099 110 110 106 205 205 197 F F F <--- 5 100-119 101 101 101 188 188 188 F F F 6 120-139 108 108 108 199 199 199 F F F 7 140-159 110 110 97 202 202 182 F F F <--- 8 160-179 107 107 107 198 198 198 F F F 9 180-199 110 110 107 205 205 198 F F F <--- 10 200-219 110 110 106 205 205 197 F F F <--- 11 220-239 110 110 106 205 205 197 F F F <--- 12 240-259 108 108 108 199 199 199 F F F 13 260-279 109 109 109 201 201 201 F F F 14 280-299 109 109 109 201 201 201 F F F 15 300-319 108 108 108 200 200 200 F F F 16 320-339 108 108 108 199 199 199 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 103 103 103 193 193 193 T T F <--- 19 380-399 105 105 105 196 196 196 T T F <--- 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 100 100 100 187 187 187 T T F <--- 23 460-479 104 104 104 194 194 194 T T F <--- 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 110 110 108 205 205 200 F F F <--- 27 540-559 110 110 108 205 205 199 T T F <--- 28 560-579 108 108 108 200 200 200 T T F <--- 29 580-599 101 101 101 188 188 188 F F F Max gen: 110, Max flush: 205 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 204 204 204 F F F 1 020-039 111 111 111 206 206 206 F F F 2 040-059 104 104 104 194 194 194 F F F 3 060-079 108 108 108 200 200 200 F F F 4 080-099 110 110 110 205 205 205 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 108 108 108 199 199 199 F F F 7 140-159 110 110 110 202 202 202 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 110 110 110 205 205 205 F F F 10 200-219 110 110 110 205 205 205 F F F 11 220-239 110 110 110 205 205 205 F F F 12 240-259 108 108 108 199 199 199 F F F 13 260-279 109 109 109 201 201 201 F F F 14 280-299 109 109 109 201 201 201 F F F 15 300-319 108 108 108 200 200 200 F F F 16 320-339 108 108 108 199 199 199 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 111 111 111 206 206 206 F F F 19 380-399 111 111 111 206 206 206 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 111 111 111 206 206 206 F F F 23 460-479 111 111 111 206 206 206 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 110 110 110 205 205 205 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 111, Max flush: 206 Begin loop 56 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.16745646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:30.18771106Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.187790657Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.205034245Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.205824595Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.205942577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.205989199Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.20604523Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"Crucible 5adc0a4b-8935-4945-83ae-f314fcea4eb4 has session id: 06a45aea-1950-44e7-a244-dccdc702a625","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.207231501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Upstairs opts: Upstairs UUID: 5adc0a4b-8935-4945-83ae-f314fcea4eb4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.207326589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Crucible stats registered with UUID: 5adc0a4b-8935-4945-83ae-f314fcea4eb4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.207373411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:30.207413139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.207828469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.20788969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.207917885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.2081789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208215573Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"5adc0a4b-8935-4945-83ae-f314fcea4eb4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208338796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208375809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208402291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208428784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208454675Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208481798Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208548951Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"ds_connection connected from Ok(127.0.0.1:47145)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208649008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"ds_connection connected from Ok(127.0.0.1:34423)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208681001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"ds_connection connected from Ok(127.0.0.1:42861)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.208755679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.211189562Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.211279058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.211527518Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212059258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212427374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[0]R flush_numbers[0..12]: [204, 206, 194, 200, 205, 188, 199, 202, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212509055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[0]R generation[0..12]: [110, 111, 104, 108, 110, 101, 108, 110, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212553994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212588993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[1]R flush_numbers[0..12]: [204, 206, 194, 200, 205, 188, 199, 202, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212621708Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[1]R generation[0..12]: [110, 111, 104, 108, 110, 101, 108, 110, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212653871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212687387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[2]R flush_numbers[0..12]: [204, 206, 194, 200, 205, 188, 199, 202, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.21271936Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[2]R generation[0..12]: [110, 111, 104, 108, 110, 101, 108, 110, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212751534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212784609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212811733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212840129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Next flush: 207","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212866771Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212893514Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"":"downstairs","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.21292206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:04:30.212947821Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"0","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.212983692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"1","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.213017449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"client":"2","":"downstairs","session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.213047157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.21307398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"5adc0a4b-8935-4945-83ae-f314fcea4eb4 is now active with session: 06a45aea-1950-44e7-a244-dccdc702a625","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.21310415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.2131298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640,"session_id":"06a45aea-1950-44e7-a244-dccdc702a625"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.213157054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} {"msg":"Activated sub_volume 5adc0a4b-8935-4945-83ae-f314fcea4eb4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:30.21318008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 327 len 24576 data: 8 9 10 12 8 7 02/30 Flush 03/30 Write block 369 len 12288 data: 5 5 6 04/30 Write block 87 len 12288 data: 10 13 12 05/30 Read block 29 len 24576 06/30 Read block 474 len 40960 07/30 Read block 197 len 8192 08/30 Read block 395 len 28672 09/30 Read block 41 len 36864 10/30 Read block 563 len 36864 11/30 Read block 117 len 16384 12/30 Read block 146 len 24576 13/30 Read block 405 len 24576 14/30 Read block 404 len 40960 15/30 Read block 291 len 40960 16/30 Write block 123 len 8192 data: 8 9 17/30 Write block 273 len 32768 data: 10 9 13 15 16 16 14 14 18/30 Read block 299 len 36864 19/30 Read block 518 len 16384 20/30 Write block 69 len 16384 data: 11 9 7 9 21/30 Read block 7 len 24576 22/30 Read block 167 len 36864 23/30 Write block 327 len 12288 data: 9 10 11 24/30 Flush 25/30 Write block 364 len 24576 data: 9 9 8 7 6 6 26/30 Write block 530 len 8192 data: 8 9 27/30 Read block 540 len 28672 28/30 Read block 239 len 28672 29/30 Write block 58 len 24576 data: 7 7 8 9 8 9 30/30 Write block 144 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:30.243790585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1640} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 204 204 204 F F F 1 020-039 111 111 111 206 206 206 F F F 2 040-059 104 104 104 194 194 194 F T T <--- 3 060-079 108 112 112 200 208 208 F T T <--- 4 080-099 110 112 112 205 208 208 F F F <--- 5 100-119 101 101 101 188 188 188 F F F 6 120-139 108 112 112 199 208 208 F F F <--- 7 140-159 110 110 110 202 202 202 F T T <--- 8 160-179 107 107 107 198 198 198 F F F 9 180-199 110 110 110 205 205 205 F F F 10 200-219 110 110 110 205 205 205 F F F 11 220-239 110 110 110 205 205 205 F F F 12 240-259 108 108 108 199 199 199 F F F 13 260-279 109 112 112 201 208 208 F F F <--- 14 280-299 109 112 112 201 208 208 F F F <--- 15 300-319 108 108 108 200 200 200 F F F 16 320-339 108 112 112 199 208 208 F F F <--- 17 340-359 102 102 102 189 189 189 F F F 18 360-379 111 112 112 206 208 208 F T T <--- 19 380-399 111 111 111 206 206 206 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 111 111 111 206 206 206 F F F 23 460-479 111 111 111 206 206 206 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 110 110 110 205 205 205 F T T <--- 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 112, Max flush: 208 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 204 204 204 F F F 1 020-039 111 111 111 206 206 206 F F F 2 040-059 113 113 113 209 209 209 F F F 3 060-079 113 113 113 209 209 209 F F F 4 080-099 112 112 112 208 208 208 F F F 5 100-119 101 101 101 188 188 188 F F F 6 120-139 112 112 112 208 208 208 F F F 7 140-159 113 113 113 209 209 209 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 110 110 110 205 205 205 F F F 10 200-219 110 110 110 205 205 205 F F F 11 220-239 110 110 110 205 205 205 F F F 12 240-259 108 108 108 199 199 199 F F F 13 260-279 112 112 112 208 208 208 F F F 14 280-299 112 112 112 208 208 208 F F F 15 300-319 108 108 108 200 200 200 F F F 16 320-339 112 112 112 208 208 208 F F F 17 340-359 102 102 102 189 189 189 F F F 18 360-379 113 113 113 209 209 209 F F F 19 380-399 111 111 111 206 206 206 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 111 111 111 206 206 206 F F F 23 460-479 111 111 111 206 206 206 F F F 24 480-499 107 107 107 198 198 198 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 113, Max flush: 209 Begin loop 57 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.784903226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.806263628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.806766811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.806840316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.806865375Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.806902448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"Crucible 94c870e7-9353-4b94-96a0-29b99914bebe has session id: b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.80810442Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Upstairs opts: Upstairs UUID: 94c870e7-9353-4b94-96a0-29b99914bebe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.80816491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Crucible stats registered with UUID: 94c870e7-9353-4b94-96a0-29b99914bebe","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.808201772Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:34.808228355Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.808738802Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809024706Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809098261Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809130505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809158911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"94c870e7-9353-4b94-96a0-29b99914bebe active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809184772Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809212888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.80934596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809408163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809439154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809470626Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809501928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46914)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809848221Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:48234)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809892378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:39539)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.809930343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.812016059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.812070727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.812525034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.812846849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.812915955Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[0]R flush_numbers[0..12]: [204, 206, 209, 209, 208, 188, 208, 209, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813089167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[0]R generation[0..12]: [110, 111, 113, 113, 112, 101, 112, 113, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813122994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813149326Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[1]R flush_numbers[0..12]: [204, 206, 209, 209, 208, 188, 208, 209, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813174896Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[1]R generation[0..12]: [110, 111, 113, 113, 112, 101, 112, 113, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813199384Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813225055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[2]R flush_numbers[0..12]: [204, 206, 209, 209, 208, 188, 208, 209, 198, 205, 205, 205]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813249994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[2]R generation[0..12]: [110, 111, 113, 113, 112, 101, 112, 113, 107, 110, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813274583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813308039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813329832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813352206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Next flush: 210","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813372626Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813393327Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"":"downstairs","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.8134151Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.8134353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"0","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813457263Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"1","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813478806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"client":"2","":"downstairs","session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813500809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813521861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"94c870e7-9353-4b94-96a0-29b99914bebe is now active with session: b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813545447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813565367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648,"session_id":"b58e6e29-cd9f-4f8a-9c53-e05dd4b31fa4"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813586358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} {"msg":"Activated sub_volume 94c870e7-9353-4b94-96a0-29b99914bebe","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:34.813604674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 116 len 40960 data: 13 11 12 10 12 10 8 9 10 8 02/30 Read block 415 len 16384 03/30 Write block 347 len 16384 data: 10 9 8 8 04/30 Read block 22 len 32768 05/30 Write block 86 len 36864 data: 13 11 14 13 14 12 13 12 11 06/30 Write block 377 len 20480 data: 13 16 19 18 20 07/30 Read block 158 len 24576 08/30 Read block 83 len 4096 09/30 Write block 181 len 12288 data: 8 10 14 10/30 Read block 169 len 12288 11/30 Read block 215 len 28672 12/30 Read block 161 len 40960 13/30 Read block 74 len 16384 14/30 Read block 18 len 28672 15/30 Read block 359 len 4096 16/30 Read block 558 len 8192 17/30 Read block 540 len 20480 18/30 Read block 278 len 24576 19/30 Write block 217 len 32768 data: 6 6 9 8 8 9 8 8 20/30 Write block 257 len 24576 data: 9 10 13 11 13 12 21/30 Read block 538 len 8192 22/30 Write block 295 len 32768 data: 11 11 12 10 9 9 7 11 23/30 Write block 327 len 20480 data: 10 11 12 13 9 24/30 Read block 324 len 12288 25/30 Read block 455 len 4096 26/30 Write block 484 len 36864 data: 16 12 8 7 7 8 7 6 6 27/30 Read block 196 len 16384 28/30 Read block 40 len 32768 29/30 Read block 297 len 4096 30/30 Write block 217 len 16384 data: 7 7 10 9 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 4 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:34.84549715Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1648} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 204 204 204 F F F 1 020-039 111 111 111 206 206 206 F F F 2 040-059 113 113 113 209 209 209 F F F 3 060-079 113 113 113 209 209 209 F F F 4 080-099 112 112 112 208 208 208 T T F <--- 5 100-119 101 101 101 188 188 188 T T T 6 120-139 112 112 112 208 208 208 T T T 7 140-159 113 113 113 209 209 209 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 110 110 110 205 205 205 T T F <--- 10 200-219 110 110 110 205 205 205 T T F <--- 11 220-239 110 110 110 205 205 205 T T F <--- 12 240-259 108 108 108 199 199 199 T T F <--- 13 260-279 112 112 112 208 208 208 T T F <--- 14 280-299 112 112 112 208 208 208 T T F <--- 15 300-319 108 108 108 200 200 200 T T F <--- 16 320-339 112 112 112 208 208 208 T T F <--- 17 340-359 102 102 102 189 189 189 T T F <--- 18 360-379 113 113 113 209 209 209 T T F <--- 19 380-399 111 111 111 206 206 206 T T F <--- 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 111 111 111 206 206 206 F F F 23 460-479 111 111 111 206 206 206 F F F 24 480-499 107 107 107 198 198 198 T T F <--- 25 500-519 107 107 107 198 198 198 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 113, Max flush: 209 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 204 204 204 F F F 1 020-039 111 111 111 206 206 206 F F F 2 040-059 113 113 113 209 209 209 F F F 3 060-079 113 113 113 209 209 209 F F F 4 080-099 114 114 114 210 210 210 F F F 5 100-119 114 114 114 210 210 210 F F F 6 120-139 114 114 114 210 210 210 F F F 7 140-159 113 113 113 209 209 209 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 114 114 114 210 210 210 F F F 10 200-219 114 114 114 210 210 210 F F F 11 220-239 114 114 114 210 210 210 F F F 12 240-259 114 114 114 210 210 210 F F F 13 260-279 114 114 114 210 210 210 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 114 114 114 210 210 210 F F F 16 320-339 114 114 114 210 210 210 F F F 17 340-359 114 114 114 210 210 210 F F F 18 360-379 114 114 114 210 210 210 F F F 19 380-399 114 114 114 210 210 210 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 108 108 108 199 199 199 F F F 22 440-459 111 111 111 206 206 206 F F F 23 460-479 111 111 111 206 206 206 F F F 24 480-499 114 114 114 210 210 210 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 114, Max flush: 210 Begin loop 58 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.432910249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.454682464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.45511627Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.455193252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.455218852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.455256036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"Crucible 416da63b-763c-436b-b236-4b40c652a72b has session id: 5a72a7ac-ec60-4c86-aa20-53b8b7a38d05","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456275988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Upstairs opts: Upstairs UUID: 416da63b-763c-436b-b236-4b40c652a72b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456337149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Crucible stats registered with UUID: 416da63b-763c-436b-b236-4b40c652a72b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456367459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:39.456393941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456750224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456803399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.456833098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457144773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"416da63b-763c-436b-b236-4b40c652a72b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.45732075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457355659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457384135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457412762Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457440196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457468702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457561335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"ds_connection connected from Ok(127.0.0.1:34373)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.45761464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"ds_connection connected from Ok(127.0.0.1:59314)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457646022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"ds_connection connected from Ok(127.0.0.1:54414)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.457682334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.459376327Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.459528306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.460185974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.460255031Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.460461608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[0]R flush_numbers[0..12]: [204, 206, 209, 209, 210, 210, 210, 209, 198, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.46103009Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[0]R generation[0..12]: [110, 111, 113, 113, 114, 114, 114, 113, 107, 114, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461073155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461115859Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[1]R flush_numbers[0..12]: [204, 206, 209, 209, 210, 210, 210, 209, 198, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461148203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[1]R generation[0..12]: [110, 111, 113, 113, 114, 114, 114, 113, 107, 114, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461180026Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461212991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[2]R flush_numbers[0..12]: [204, 206, 209, 209, 210, 210, 210, 209, 198, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461244373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[2]R generation[0..12]: [110, 111, 113, 113, 114, 114, 114, 113, 107, 114, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461275625Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.46130868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Max found gen is 115","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461335663Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Generation requested: 116 >= found:115","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461363899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Next flush: 211","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461390291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461417014Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"":"downstairs","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461444738Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461470259Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"0","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:04:39.461498835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"1","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461535157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"client":"2","":"downstairs","session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461563883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461590696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"416da63b-763c-436b-b236-4b40c652a72b is now active with session: 5a72a7ac-ec60-4c86-aa20-53b8b7a38d05","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461620726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461645895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656,"session_id":"5a72a7ac-ec60-4c86-aa20-53b8b7a38d05"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461672528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} {"msg":"Activated sub_volume 416da63b-763c-436b-b236-4b40c652a72b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:39.461695413Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 190 len 12288 data: 9 8 8 02/30 Read block 102 len 16384 03/30 Write block 13 len 24576 data: 15 13 12 10 10 9 04/30 Write block 435 len 36864 data: 10 11 12 12 9 8 11 10 10 05/30 Flush 06/30 Write block 475 len 24576 data: 12 11 11 12 10 10 07/30 Flush 08/30 Write block 317 len 12288 data: 7 8 7 09/30 Read block 138 len 32768 10/30 Read block 109 len 36864 11/30 Read block 351 len 36864 12/30 Read block 582 len 8192 13/30 Read block 579 len 24576 14/30 Read block 571 len 24576 15/30 Write block 137 len 20480 data: 4 3 5 6 6 16/30 Write block 110 len 40960 data: 11 10 13 13 12 13 14 12 13 11 17/30 Flush 18/30 Write block 65 len 36864 data: 8 10 11 14 12 10 8 10 10 19/30 Read block 224 len 4096 20/30 Read block 374 len 28672 21/30 Read block 199 len 36864 22/30 Read block 221 len 16384 23/30 Read block 474 len 28672 24/30 Write block 342 len 20480 data: 8 6 6 6 9 25/30 Write block 37 len 36864 data: 13 13 9 7 5 5 7 7 9 26/30 Read block 584 len 16384 27/30 Read block 179 len 28672 28/30 Read block 412 len 24576 29/30 Read block 77 len 24576 30/30 Write block 429 len 16384 data: 9 10 9 6 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Read 9 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 25 27 Done 24 24 0 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) Some(JobId(1004)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:39.494528525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1656} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 111 111 111 206 206 206 T T F <--- 2 040-059 113 113 113 209 209 209 T T F <--- 3 060-079 113 113 113 209 209 209 T T F <--- 4 080-099 114 114 114 210 210 210 F F F 5 100-119 116 116 114 213 213 210 F F F <--- 6 120-139 116 116 114 213 213 210 F F F <--- 7 140-159 116 116 113 213 213 209 F F F <--- 8 160-179 107 107 107 198 198 198 F F F 9 180-199 116 116 116 211 211 211 F F F 10 200-219 114 114 114 210 210 210 F F F 11 220-239 114 114 114 210 210 210 F F F 12 240-259 114 114 114 210 210 210 F F F 13 260-279 114 114 114 210 210 210 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 116 116 114 213 213 210 F F F <--- 16 320-339 114 114 114 210 210 210 F F F 17 340-359 114 114 114 210 210 210 T T F <--- 18 360-379 114 114 114 210 210 210 F F F 19 380-399 114 114 114 210 210 210 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 116 116 116 211 211 211 T T F <--- 22 440-459 116 116 116 211 211 211 F F F 23 460-479 116 116 111 212 212 206 F F F <--- 24 480-499 116 116 114 212 212 210 F F F <--- 25 500-519 107 107 107 198 198 198 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 116, Max flush: 213 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 117 117 117 214 214 214 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 117 117 117 214 214 214 F F F 4 080-099 114 114 114 210 210 210 F F F 5 100-119 116 116 116 213 213 213 F F F 6 120-139 116 116 116 213 213 213 F F F 7 140-159 116 116 116 213 213 213 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 116 116 116 211 211 211 F F F 10 200-219 114 114 114 210 210 210 F F F 11 220-239 114 114 114 210 210 210 F F F 12 240-259 114 114 114 210 210 210 F F F 13 260-279 114 114 114 210 210 210 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 116 116 116 213 213 213 F F F 16 320-339 114 114 114 210 210 210 F F F 17 340-359 117 117 117 214 214 214 F F F 18 360-379 114 114 114 210 210 210 F F F 19 380-399 114 114 114 210 210 210 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 116 116 116 212 212 212 F F F 24 480-499 116 116 116 212 212 212 F F F 25 500-519 107 107 107 198 198 198 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 117, Max flush: 214 Begin loop 59 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.034877819Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.058271463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.058707062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.058786139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.05881246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.058851568Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"Crucible 1d141c01-e40c-4e44-88f0-f522d2d29846 has session id: ddebf709-6a20-4f90-85af-db21c9041ef1","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.059715322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Upstairs opts: Upstairs UUID: 1d141c01-e40c-4e44-88f0-f522d2d29846, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.05978001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Crucible stats registered with UUID: 1d141c01-e40c-4e44-88f0-f522d2d29846","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.05981051Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:44.059846331Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060158136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060216231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060270337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060375665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060419221Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"1d141c01-e40c-4e44-88f0-f522d2d29846 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060455813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060492646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060534098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060580049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060625759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060670858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.06071756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"ds_connection connected from Ok(127.0.0.1:58474)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060766306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"ds_connection connected from Ok(127.0.0.1:45811)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.060814451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"ds_connection connected from Ok(127.0.0.1:42940)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.061278126Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"io task","client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.062848224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.063094551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.063136333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.063830142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.063959848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[0]R flush_numbers[0..12]: [211, 214, 214, 214, 210, 213, 213, 213, 198, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064168931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[0]R generation[0..12]: [116, 117, 117, 117, 114, 116, 116, 116, 107, 116, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064201064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064227807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[1]R flush_numbers[0..12]: [211, 214, 214, 214, 210, 213, 213, 213, 198, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064253538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[1]R generation[0..12]: [116, 117, 117, 117, 114, 116, 116, 116, 107, 116, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064279018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.06430524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[2]R flush_numbers[0..12]: [211, 214, 214, 214, 210, 213, 213, 213, 198, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064330099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[2]R generation[0..12]: [116, 117, 117, 117, 114, 116, 116, 116, 107, 116, 114, 114]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064355188Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.06438142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064407341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064430407Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Next flush: 215","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064451739Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064473231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"":"downstairs","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064495375Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064516076Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"0","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.06453844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"1","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064560433Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"client":"2","":"downstairs","session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064582958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.06460428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"1d141c01-e40c-4e44-88f0-f522d2d29846 is now active with session: ddebf709-6a20-4f90-85af-db21c9041ef1","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064628307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064648447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664,"session_id":"ddebf709-6a20-4f90-85af-db21c9041ef1"} {"msg":"Wait for a query_work_queue command to finish before sending IO The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064678085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} {"msg":"Activated sub_volume 1d141c01-e40c-4e44-88f0-f522d2d29846","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:44.064696622Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 67 len 8192 data: 12 15 02/30 Read block 241 len 28672 03/30 Read block 130 len 40960 04/30 Read block 407 len 36864 05/30 Read block 31 len 28672 06/30 Write block 363 len 28672 data: 7 10 10 9 8 7 7 07/30 Read block 469 len 8192 08/30 Read block 46 len 28672 09/30 Write block 256 len 28672 data: 9 10 11 14 12 14 13 10/30 Write block 218 len 20480 data: 8 11 10 9 10 11/30 Read block 449 len 8192 12/30 Read block 103 len 16384 13/30 Read block 213 len 36864 14/30 Write block 470 len 20480 data: 8 10 9 10 10 15/30 Flush 16/30 Flush 17/30 Write block 30 len 24576 data: 11 8 11 11 13 13 18/30 Read block 412 len 20480 19/30 Write block 150 len 24576 data: 8 9 8 9 7 7 20/30 Write block 270 len 16384 data: 5 5 9 11 21/30 Read block 25 len 8192 22/30 Read block 37 len 12288 23/30 Write block 318 len 20480 data: 9 8 8 8 7 24/30 Read block 371 len 12288 25/30 Read block 192 len 4096 26/30 Write block 510 len 8192 data: 12 11 27/30 Read block 466 len 16384 28/30 Read block 70 len 4096 29/30 Write block 262 len 16384 data: 14 10 9 8 30/30 Read block 435 len 32768 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:44.096862918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1664} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 117 117 117 214 214 214 F T T <--- 2 040-059 117 117 117 214 214 214 F F F 3 060-079 117 118 118 214 215 215 T F F <--- 4 080-099 114 114 114 210 210 210 F F F 5 100-119 116 116 116 213 213 213 F F F 6 120-139 116 116 116 213 213 213 F F F 7 140-159 116 116 116 213 213 213 F T T <--- 8 160-179 107 107 107 198 198 198 F F F 9 180-199 116 116 116 211 211 211 F F F 10 200-219 114 118 118 210 215 215 F F F <--- 11 220-239 114 118 118 210 215 215 F F F <--- 12 240-259 114 118 118 210 215 215 F F F <--- 13 260-279 114 118 118 210 215 215 F T T <--- 14 280-299 114 114 114 210 210 210 F F F 15 300-319 116 116 116 213 213 213 F T T <--- 16 320-339 114 114 114 210 210 210 F T T <--- 17 340-359 117 117 117 214 214 214 F F F 18 360-379 114 118 118 210 215 215 F F F <--- 19 380-399 114 114 114 210 210 210 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 116 118 118 212 215 215 F F F <--- 24 480-499 116 116 116 212 212 212 F F F 25 500-519 107 107 107 198 198 198 F T T <--- 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 118, Max flush: 215 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 119 119 119 216 216 216 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 118 118 118 215 215 215 F F F 4 080-099 114 114 114 210 210 210 F F F 5 100-119 116 116 116 213 213 213 F F F 6 120-139 116 116 116 213 213 213 F F F 7 140-159 119 119 119 216 216 216 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 116 116 116 211 211 211 F F F 10 200-219 118 118 118 215 215 215 F F F 11 220-239 118 118 118 215 215 215 F F F 12 240-259 118 118 118 215 215 215 F F F 13 260-279 119 119 119 216 216 216 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 119 119 119 216 216 216 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 117 117 117 214 214 214 F F F 18 360-379 118 118 118 215 215 215 F F F 19 380-399 114 114 114 210 210 210 F F F 20 400-419 106 106 106 197 197 197 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 118 118 118 215 215 215 F F F 24 480-499 116 116 116 212 212 212 F F F 25 500-519 119 119 119 216 216 216 F F F 26 520-539 113 113 113 209 209 209 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 F F F Max gen: 119, Max flush: 216 Begin loop 60 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.655336188Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.677961055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.678530289Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.678600868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.678635526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.678683922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"Crucible 6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8 has session id: 69acf447-fc44-4ea2-953c-533f2037e5ba","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.679660459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Upstairs opts: Upstairs UUID: 6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.679720147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Crucible stats registered with UUID: 6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.679748863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:48.679773632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680259681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680372033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680402654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680594673Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680630824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680754257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680784467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680810829Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680843403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680869725Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680896959Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.680992147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:58022)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.681024531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:35411)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.681052125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:49109)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.681134217Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"io task","client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.683190325Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.683430318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.68363366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.683911949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684245828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[0]R flush_numbers[0..12]: [211, 216, 214, 215, 210, 213, 213, 216, 198, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684648011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[0]R generation[0..12]: [116, 119, 117, 118, 114, 116, 116, 119, 107, 116, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684686337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684719943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[1]R flush_numbers[0..12]: [211, 216, 214, 215, 210, 213, 213, 216, 198, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.68475873Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[1]R generation[0..12]: [116, 119, 117, 118, 114, 116, 116, 119, 107, 116, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684791114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.68482452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[2]R flush_numbers[0..12]: [211, 216, 214, 215, 210, 213, 213, 216, 198, 211, 215, 215]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684856553Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[2]R generation[0..12]: [116, 119, 117, 118, 114, 116, 116, 119, 107, 116, 118, 118]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684887895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.68492084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684947813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.684975948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Next flush: 217","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685002381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685043371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"":"downstairs","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685072218Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.68509822Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"0","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2026-03-06T06:04:48.685139581Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"1","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685168579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"client":"2","":"downstairs","session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685200912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685227996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8 is now active with session: 69acf447-fc44-4ea2-953c-533f2037e5ba","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685258175Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685283245Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672,"session_id":"69acf447-fc44-4ea2-953c-533f2037e5ba"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685308164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} {"msg":"Activated sub_volume 6f8bcdab-ffb2-45f0-adcb-a34dd2f405a8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:48.685329456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 77 len 24576 data: 15 12 11 12 15 18 02/30 Read block 243 len 32768 03/30 Read block 175 len 4096 04/30 Read block 181 len 4096 05/30 Read block 402 len 36864 06/30 Flush 07/30 Write block 124 len 28672 data: 11 9 8 10 7 5 10 08/30 Read block 436 len 16384 09/30 Read block 455 len 40960 10/30 Write block 537 len 8192 data: 12 11 11/30 Read block 313 len 20480 12/30 Read block 469 len 12288 13/30 Write block 407 len 24576 data: 7 9 8 9 10 10 14/30 Flush 15/30 Write block 28 len 16384 data: 11 14 12 9 16/30 Read block 553 len 28672 17/30 Read block 81 len 16384 18/30 Write block 198 len 16384 data: 9 7 7 6 19/30 Read block 215 len 36864 20/30 Read block 555 len 36864 21/30 Write block 373 len 36864 data: 9 10 11 10 14 17 20 19 21 22/30 Write block 494 len 20480 data: 8 7 9 10 11 23/30 Write block 119 len 24576 data: 12 13 11 9 10 12 24/30 Write block 587 len 36864 data: 6 6 6 6 4 4 6 4 4 25/30 Read block 484 len 24576 26/30 Read block 141 len 16384 27/30 Read block 144 len 12288 28/30 Write block 399 len 32768 data: 12 12 14 13 16 13 13 10 29/30 Write block 225 len 32768 data: 9 9 8 9 7 7 8 8 30/30 Read block 61 len 8192 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 1 Done Done Done false 1004 Acked Read 9 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 8 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 26 26 Done 30 30 4 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:48.717184868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1672} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 119 119 119 216 216 216 T T F <--- 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 118 217 217 215 F F T <--- 4 080-099 120 120 114 217 217 210 F F T <--- 5 100-119 116 116 116 213 213 213 T T F <--- 6 120-139 120 120 116 218 218 213 T T F <--- 7 140-159 119 119 119 216 216 216 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 116 116 116 211 211 211 T T F <--- 10 200-219 118 118 118 215 215 215 T T F <--- 11 220-239 118 118 118 215 215 215 T T F <--- 12 240-259 118 118 118 215 215 215 F F F 13 260-279 119 119 119 216 216 216 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 119 119 119 216 216 216 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 117 117 117 214 214 214 F F F 18 360-379 118 118 118 215 215 215 T T F <--- 19 380-399 114 114 114 210 210 210 T T F <--- 20 400-419 120 120 106 218 218 197 T T F <--- 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 118 118 118 215 215 215 F F F 24 480-499 116 116 116 212 212 212 T T F <--- 25 500-519 119 119 119 216 216 216 F F F 26 520-539 120 120 113 218 218 209 F F F <--- 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 101 101 101 188 188 188 T T F <--- Max gen: 120, Max flush: 218 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 121 121 121 219 219 219 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 120 120 120 217 217 217 F F F 5 100-119 121 121 121 219 219 219 F F F 6 120-139 121 121 121 219 219 219 F F F 7 140-159 119 119 119 216 216 216 F F F 8 160-179 107 107 107 198 198 198 F F F 9 180-199 121 121 121 219 219 219 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 121 121 121 219 219 219 F F F 12 240-259 118 118 118 215 215 215 F F F 13 260-279 119 119 119 216 216 216 F F F 14 280-299 114 114 114 210 210 210 F F F 15 300-319 119 119 119 216 216 216 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 117 117 117 214 214 214 F F F 18 360-379 121 121 121 219 219 219 F F F 19 380-399 121 121 121 219 219 219 F F F 20 400-419 121 121 121 219 219 219 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 118 118 118 215 215 215 F F F 24 480-499 121 121 121 219 219 219 F F F 25 500-519 119 119 119 216 216 216 F F F 26 520-539 120 120 120 218 218 218 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 121 121 121 219 219 219 F F F Max gen: 121, Max flush: 219 Begin loop 61 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.299677708Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:53.316280111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.316361732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.332253612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33282008Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.332882834Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.332908114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.332938825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"Crucible 1f924cfc-729f-44da-90a8-84eecdd581e4 has session id: 0768127f-613a-4994-b352-c2e92bbc420d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334101178Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Upstairs opts: Upstairs UUID: 1f924cfc-729f-44da-90a8-84eecdd581e4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334176247Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Crucible stats registered with UUID: 1f924cfc-729f-44da-90a8-84eecdd581e4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334209111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:53.334231826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334660101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33473565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334766381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334937258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.334969131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"1f924cfc-729f-44da-90a8-84eecdd581e4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335069479Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335218453Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33525752Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335755072Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335801674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335831323Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335859999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"ds_connection connected from Ok(127.0.0.1:33340)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335890469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"ds_connection connected from Ok(127.0.0.1:44947)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335920799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"ds_connection connected from Ok(127.0.0.1:38569)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.335974926Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"io task","client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.337399607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.337762473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.337802472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338408558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338561931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[0]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 219, 216, 198, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338777486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[0]R generation[0..12]: [116, 121, 117, 120, 120, 121, 121, 119, 107, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338811403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338837094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[1]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 219, 216, 198, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338861322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[1]R generation[0..12]: [116, 121, 117, 120, 120, 121, 121, 119, 107, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338885059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338908826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[2]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 219, 216, 198, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338931981Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[2]R generation[0..12]: [116, 121, 117, 120, 120, 121, 121, 119, 107, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338963944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.338988964Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339009204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339031327Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Next flush: 220","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339051197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339071477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"":"downstairs","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339092388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339111596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"0","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339132838Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"1","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339153328Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"client":"2","":"downstairs","session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33917475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33919458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"1f924cfc-729f-44da-90a8-84eecdd581e4 is now active with session: 0768127f-613a-4994-b352-c2e92bbc420d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339216423Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339234909Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680,"session_id":"0768127f-613a-4994-b352-c2e92bbc420d"} Wait for a query_work_queue command to finish before sending IO {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.33926598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} {"msg":"Activated sub_volume 1f924cfc-729f-44da-90a8-84eecdd581e4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:53.339284677Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 290 len 32768 data: 11 11 10 8 8 12 12 13 02/30 Read block 583 len 4096 03/30 Flush 04/30 Write block 387 len 12288 data: 11 11 7 05/30 Read block 365 len 20480 06/30 Write block 348 len 40960 data: 10 9 9 7 6 7 5 7 9 6 07/30 Read block 300 len 4096 08/30 Read block 155 len 20480 09/30 Read block 85 len 24576 10/30 Read block 313 len 20480 11/30 Write block 278 len 16384 data: 17 15 15 15 12/30 Read block 416 len 36864 13/30 Write block 535 len 20480 data: 8 10 13 12 13 14/30 Read block 587 len 32768 15/30 Read block 534 len 8192 16/30 Read block 102 len 8192 17/30 Write block 463 len 4096 data: 8 18/30 Flush 19/30 Write block 167 len 12288 data: 10 9 9 20/30 Write block 494 len 20480 data: 9 8 10 11 12 21/30 Read block 76 len 4096 22/30 Flush 23/30 Write block 463 len 16384 data: 9 8 10 8 24/30 Read block 247 len 20480 25/30 Write block 446 len 12288 data: 12 10 11 26/30 Read block 186 len 36864 27/30 Write block 315 len 8192 data: 11 9 28/30 Read block 46 len 8192 29/30 Write block 120 len 40960 data: 14 12 10 11 13 10 9 11 8 6 30/30 Read block 226 len 24576 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 10 Sent Done Done false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:53.369675179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1680} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 121 121 121 219 219 219 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 120 120 120 217 217 217 F F F 5 100-119 121 121 121 219 219 219 F F F 6 120-139 121 121 121 219 219 219 F T T <--- 7 140-159 119 119 119 216 216 216 F F F 8 160-179 107 122 122 198 222 222 F F F <--- 9 180-199 121 121 121 219 219 219 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 121 121 121 219 219 219 F F F 12 240-259 118 118 118 215 215 215 F F F 13 260-279 119 122 122 216 221 221 F F F <--- 14 280-299 114 122 122 210 221 221 T F F <--- 15 300-319 119 119 119 216 216 216 F T T <--- 16 320-339 119 119 119 216 216 216 F F F 17 340-359 117 122 122 214 221 221 F F F <--- 18 360-379 121 121 121 219 219 219 F F F 19 380-399 121 122 122 219 221 221 F F F <--- 20 400-419 121 121 121 219 219 219 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 116 116 116 211 211 211 F T T <--- 23 460-479 118 122 122 215 221 221 F T T <--- 24 480-499 121 122 122 219 222 222 F F F <--- 25 500-519 119 119 119 216 216 216 F F F 26 520-539 120 122 122 218 221 221 F F F <--- 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 121 121 121 219 219 219 F F F Max gen: 122, Max flush: 222 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 211 211 211 F F F 1 020-039 121 121 121 219 219 219 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 120 120 120 217 217 217 F F F 5 100-119 121 121 121 219 219 219 F F F 6 120-139 123 123 123 223 223 223 F F F 7 140-159 119 119 119 216 216 216 F F F 8 160-179 122 122 122 222 222 222 F F F 9 180-199 121 121 121 219 219 219 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 121 121 121 219 219 219 F F F 12 240-259 118 118 118 215 215 215 F F F 13 260-279 122 122 122 221 221 221 F F F 14 280-299 122 122 122 221 221 221 F F F 15 300-319 123 123 123 223 223 223 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 122 122 122 221 221 221 F F F 18 360-379 121 121 121 219 219 219 F F F 19 380-399 122 122 122 221 221 221 F F F 20 400-419 121 121 121 219 219 219 F F F 21 420-439 117 117 117 214 214 214 F F F 22 440-459 123 123 123 223 223 223 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 122 122 122 222 222 222 F F F 25 500-519 119 119 119 216 216 216 F F F 26 520-539 122 122 122 221 221 221 F F F 27 540-559 111 111 111 206 206 206 F F F 28 560-579 111 111 111 206 206 206 F F F 29 580-599 121 121 121 219 219 219 F F F Max gen: 123, Max flush: 223 Begin loop 62 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.903113125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.924899969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.925447078Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.925521666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.925546164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.925582896Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"Crucible 7a3d64cd-f1b2-4246-b9a7-17633c0d79b3 has session id: afa8009d-6500-438a-9152-b8b4fcc48614","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.926680833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Upstairs opts: Upstairs UUID: 7a3d64cd-f1b2-4246-b9a7-17633c0d79b3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.926742725Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Crucible stats registered with UUID: 7a3d64cd-f1b2-4246-b9a7-17633c0d79b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.926772644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:57.926799237Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927239505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927349472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927381656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927509128Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927541081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"7a3d64cd-f1b2-4246-b9a7-17633c0d79b3 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927660576Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927703221Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927732378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927760604Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.927788188Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.928035015Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.928108631Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"ds_connection connected from Ok(127.0.0.1:59231)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.928140433Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"ds_connection connected from Ok(127.0.0.1:40637)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.928171765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"ds_connection connected from Ok(127.0.0.1:49897)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.928227686Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"io task","client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.930140861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.930192894Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.9306319Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.930987602Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.931066848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[0]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 223, 216, 222, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.93176727Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[0]R generation[0..12]: [116, 121, 117, 120, 120, 121, 123, 119, 122, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.931864973Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.931903169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[1]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 223, 216, 222, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.931938258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[1]R generation[0..12]: [116, 121, 117, 120, 120, 121, 123, 119, 122, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.931972766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932008837Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[2]R flush_numbers[0..12]: [211, 219, 214, 217, 217, 219, 223, 216, 222, 219, 219, 219]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932043455Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[2]R generation[0..12]: [116, 121, 117, 120, 120, 121, 123, 119, 122, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932077773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932113624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932143052Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932176688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Next flush: 224","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932206006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932235033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"":"downstairs","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932265564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932293248Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"0","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Transition from WaitQuorum to Active","vWait for a query_work_queue command to finish before sending IO ":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.93233981Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"1","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932371773Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"client":"2","":"downstairs","session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932402915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932432112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"7a3d64cd-f1b2-4246-b9a7-17633c0d79b3 is now active with session: afa8009d-6500-438a-9152-b8b4fcc48614","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932464897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932492572Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688,"session_id":"afa8009d-6500-438a-9152-b8b4fcc48614"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932521759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} {"msg":"Activated sub_volume 7a3d64cd-f1b2-4246-b9a7-17633c0d79b3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:04:57.932546889Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 351 len 24576 data: 8 7 8 6 8 10 02/30 Write block 588 len 28672 data: 7 7 7 5 5 7 5 03/30 Read block 144 len 32768 04/30 Write block 196 len 4096 data: 10 05/30 Write block 261 len 36864 data: 15 15 11 10 9 9 9 8 6 06/30 Flush 07/30 Write block 583 len 28672 data: 7 7 6 5 7 8 8 08/30 Flush 09/30 Write block 137 len 32768 data: 5 4 6 7 7 6 7 10 10/30 Flush 11/30 Write block 83 len 4096 data: 16 12/30 Read block 101 len 8192 13/30 Write block 569 len 12288 data: 12 13 11 14/30 Write block 379 len 36864 data: 21 20 22 18 17 17 15 13 12 15/30 Read block 159 len 4096 16/30 Write block 8 len 36864 data: 9 11 14 13 14 16 14 13 11 17/30 Write block 220 len 32768 data: 11 10 11 9 9 10 10 9 18/30 Read block 573 len 8192 19/30 Read block 286 len 12288 20/30 Write block 441 len 32768 data: 12 11 11 9 12 13 11 12 21/30 Flush 22/30 Write block 557 len 24576 data: 8 9 9 6 5 7 23/30 Read block 344 len 28672 24/30 Read block 464 len 8192 25/30 Write block 549 len 20480 data: 13 11 10 11 9 26/30 Write block 273 len 4096 data: 12 27/30 Write block 434 len 40960 data: 9 11 12 13 13 10 9 13 12 12 28/30 Write block 543 len 8192 data: 10 9 29/30 Read block 107 len 16384 30/30 Read block 177 len 28672 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 7 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 25 25 Done 30 30 5 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:04:57.963755084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1688} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 116 227 227 211 F F F <--- 1 020-039 121 121 121 219 219 219 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 124 124 120 227 227 217 F F F <--- 5 100-119 121 121 121 219 219 219 F F F 6 120-139 124 124 123 226 226 223 F F F <--- 7 140-159 124 124 119 226 226 216 F F F <--- 8 160-179 122 122 122 222 222 222 F F F 9 180-199 124 124 121 224 224 219 F F T <--- 10 200-219 121 121 121 219 219 219 F F F 11 220-239 124 124 121 227 227 219 F F F <--- 12 240-259 118 118 118 215 215 215 F F F 13 260-279 124 124 122 224 224 221 T T T <--- 14 280-299 122 122 122 221 221 221 F F F 15 300-319 123 123 123 223 223 223 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 122 224 224 221 F F T <--- 18 360-379 124 124 121 227 227 219 F F F <--- 19 380-399 124 124 122 227 227 221 F F F <--- 20 400-419 121 121 121 219 219 219 F F F 21 420-439 117 117 117 214 214 214 T T F <--- 22 440-459 124 124 123 227 227 223 T T F <--- 23 460-479 123 123 123 223 223 223 F F F 24 480-499 122 122 122 222 222 222 F F F 25 500-519 119 119 119 216 216 216 F F F 26 520-539 122 122 122 221 221 221 F F F 27 540-559 111 111 111 206 206 206 T T F <--- 28 560-579 124 124 111 227 227 206 T T F <--- 29 580-599 124 124 121 225 225 219 F F T <--- Max gen: 124, Max flush: 227 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 124 227 227 227 F F F 1 020-039 121 121 121 219 219 219 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 124 124 124 227 227 227 F F F 5 100-119 121 121 121 219 219 219 F F F 6 120-139 124 124 124 226 226 226 F F F 7 140-159 124 124 124 226 226 226 F F F 8 160-179 122 122 122 222 222 222 F F F 9 180-199 124 124 124 224 224 224 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 124 124 124 227 227 227 F F F 12 240-259 118 118 118 215 215 215 F F F 13 260-279 125 125 125 228 228 228 F F F 14 280-299 122 122 122 221 221 221 F F F 15 300-319 123 123 123 223 223 223 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 124 224 224 224 F F F 18 360-379 124 124 124 227 227 227 F F F 19 380-399 124 124 124 227 227 227 F F F 20 400-419 121 121 121 219 219 219 F F F 21 420-439 125 125 125 228 228 228 F F F 22 440-459 125 125 125 228 228 228 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 122 122 122 222 222 222 F F F 25 500-519 119 119 119 216 216 216 F F F 26 520-539 122 122 122 221 221 221 F F F 27 540-559 125 125 125 228 228 228 F F F 28 560-579 125 125 125 228 228 228 F F F 29 580-599 124 124 124 225 225 225 F F F Max gen: 125, Max flush: 228 Begin loop 63 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.57685342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.593906572Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.594438642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.594550102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.594576033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.594613006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"Crucible 5126c5b4-febe-4424-93dc-c236bf54d754 has session id: 48b79c58-f0d0-463b-9823-0e4481ec45b9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.595787343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Upstairs opts: Upstairs UUID: 5126c5b4-febe-4424-93dc-c236bf54d754, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.595874756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Crucible stats registered with UUID: 5126c5b4-febe-4424-93dc-c236bf54d754","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.595906358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:02.595939513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596412196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596501292Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596531963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596796444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596829139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"5126c5b4-febe-4424-93dc-c236bf54d754 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596931421Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.59696129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.596989656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597018723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.59704742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597076176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597348304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53659)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597424033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43752)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597482358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:02.59753923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:02.597593567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:02.597648887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597688094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.597757611Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.599454971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.59982938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.600113951Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:02.600535503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.599765911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:48400)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.599903392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"io task","client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.602468414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[0]R flush_numbers[0..12]: [227, 219, 214, 217, 227, 219, 226, 226, 222, 224, 219, 227]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603397467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[0]R generation[0..12]: [124, 121, 117, 120, 124, 121, 124, 124, 122, 124, 121, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603514788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.60355104Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[1]R flush_numbers[0..12]: [227, 219, 214, 217, 227, 219, 226, 226, 222, 224, 219, 227]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603583314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[1]R generation[0..12]: [124, 121, 117, 120, 124, 121, 124, 124, 122, 124, 121, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603614946Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603647841Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[2]R flush_numbers[0..12]: [227, 219, 214, 217, 227, 219, 226, 226, 222, 224, 219, 227]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603679544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[2]R generation[0..12]: [124, 121, 117, 120, 124, 121, 124, 124, 122, 124, 121, 124]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603711316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603744241Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603772126Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603800532Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Next flush: 229","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603826974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603853888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"":"downstairs","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603886632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603912143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"0","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603951821Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"1","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.603984756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"client":"2","":"downstairs","session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604013613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604040616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"5126c5b4-febe-4424-93dc-c236bf54d754 is now active with session: 48b79c58-f0d0-463b-9823-0e4481ec45b9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604070756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604095985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696,"session_id":"48b79c58-f0d0-463b-9823-0e4481ec45b9"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604122969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} {"msg":"Activated sub_volume 5126c5b4-febe-4424-93dc-c236bf54d754","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:12.604146305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 297 len 28672 data: 14 11 10 10 8 12 11 02/30 Read block 63 len 40960 03/30 Read block 136 len 20480 04/30 Read block 543 len 16384 05/30 Read block 119 len 40960 06/30 Write block 362 len 36864 data: 8 8 11 11 10 9 8 8 6 07/30 Write block 130 len 32768 data: 11 8 7 7 5 5 3 6 08/30 Read block 102 len 16384 09/30 Write block 494 len 8192 data: 10 9 10/30 Read block 188 len 4096 11/30 Write block 313 len 20480 data: 12 11 12 10 8 12/30 Write block 16 len 32768 data: 12 11 10 8 8 9 5 6 13/30 Flush 14/30 Write block 99 len 32768 data: 5 6 7 7 8 9 9 8 15/30 Write block 224 len 28672 data: 10 11 11 10 10 8 8 16/30 Write block 452 len 4096 data: 9 17/30 Write block 458 len 8192 data: 12 8 18/30 Read block 22 len 36864 19/30 Flush 20/30 Write block 81 len 28672 data: 16 19 17 14 15 14 12 21/30 Read block 329 len 32768 22/30 Write block 518 len 24576 data: 7 11 12 12 11 10 23/30 Write block 394 len 40960 data: 6 4 4 6 8 13 13 15 14 17 24/30 Read block 270 len 20480 25/30 Read block 388 len 28672 26/30 Write block 240 len 40960 data: 12 12 14 15 13 13 11 10 11 8 27/30 Write block 576 len 4096 data: 9 28/30 Write block 296 len 20480 data: 13 15 12 11 11 29/30 Write block 154 len 24576 data: 8 8 9 9 6 6 30/30 Read block 523 len 24576 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Read 10 Done Sent Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 6 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:12.636546542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1696} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 124 126 229 227 229 F F F <--- 1 020-039 126 121 126 229 219 229 F F F <--- 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 126 124 126 230 227 230 T F T <--- 5 100-119 126 121 126 230 219 230 F F F <--- 6 120-139 126 124 126 229 226 229 F F F <--- 7 140-159 124 124 124 226 226 226 T F T <--- 8 160-179 122 122 122 222 222 222 F F F 9 180-199 124 124 124 224 224 224 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 124 126 230 227 230 F F F <--- 12 240-259 118 118 118 215 215 215 T F T <--- 13 260-279 125 125 125 228 228 228 F F F 14 280-299 126 122 126 229 221 229 T F T <--- 15 300-319 126 123 126 229 223 229 T F T <--- 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 124 224 224 224 F F F 18 360-379 126 124 126 229 227 229 F F F <--- 19 380-399 124 124 124 227 227 227 T F T <--- 20 400-419 121 121 121 219 219 219 T F T <--- 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 125 126 230 228 230 F F F <--- 23 460-479 123 123 123 223 223 223 F F F 24 480-499 126 122 126 229 222 229 F F F <--- 25 500-519 119 119 119 216 216 216 T F T <--- 26 520-539 122 122 122 221 221 221 T F T <--- 27 540-559 125 125 125 228 228 228 F F F 28 560-579 125 125 125 228 228 228 T F T <--- 29 580-599 124 124 124 225 225 225 F F F Max gen: 126, Max flush: 230 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 229 229 229 F F F 1 020-039 126 126 126 229 229 229 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 127 127 127 231 231 231 F F F 5 100-119 126 126 126 230 230 230 F F F 6 120-139 126 126 126 229 229 229 F F F 7 140-159 127 127 127 231 231 231 F F F 8 160-179 122 122 122 222 222 222 F F F 9 180-199 124 124 124 224 224 224 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 126 126 230 230 230 F F F 12 240-259 127 127 127 231 231 231 F F F 13 260-279 125 125 125 228 228 228 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 127 127 127 231 231 231 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 124 224 224 224 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 127 127 127 231 231 231 F F F 20 400-419 127 127 127 231 231 231 F F F 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 126 126 230 230 230 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 126 126 126 229 229 229 F F F 25 500-519 127 127 127 231 231 231 F F F 26 520-539 127 127 127 231 231 231 F F F 27 540-559 125 125 125 228 228 228 F F F 28 560-579 127 127 127 231 231 231 F F F 29 580-599 124 124 124 225 225 225 F F F Max gen: 127, Max flush: 231 Begin loop 64 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.236818588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.256171147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.256705872Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.256805469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.256828003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.256860347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"Crucible ba3a80c4-b64e-4e82-b8a9-41069a543d2d has session id: 18b4b962-2363-41aa-ad47-dac38616a44d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.258241723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Upstairs opts: Upstairs UUID: ba3a80c4-b64e-4e82-b8a9-41069a543d2d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.258315258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Crucible stats registered with UUID: ba3a80c4-b64e-4e82-b8a9-41069a543d2d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.25834664Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:17.258368563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.258756709Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.258801768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.258824774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259165787Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259238831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"ba3a80c4-b64e-4e82-b8a9-41069a543d2d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259348016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259373216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259394618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.25941576Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259436842Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259472372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.259507411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"ds_connection connected from Ok(127.0.0.1:51254)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.26019117Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"ds_connection connected from Ok(127.0.0.1:44870)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.260254084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"ds_connection connected from Ok(127.0.0.1:41193)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.260280567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"io task","client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.261776669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.261976353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262152791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262368948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262751112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[0]R flush_numbers[0..12]: [229, 229, 214, 217, 231, 230, 229, 231, 222, 224, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262930436Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[0]R generation[0..12]: [126, 126, 117, 120, 127, 126, 126, 127, 122, 124, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262960295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.262985735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[1]R flush_numbers[0..12]: [229, 229, 214, 217, 231, 230, 229, 231, 222, 224, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263034932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[1]R generation[0..12]: [126, 126, 117, 120, 127, 126, 126, 127, 122, 124, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.26305914Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263084189Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[2]R flush_numbers[0..12]: [229, 229, 214, 217, 231, 230, 229, 231, 222, 224, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263108056Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[2]R generation[0..12]: [126, 126, 117, 120, 127, 126, 126, 127, 122, 124, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263137915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263163335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263184487Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263205559Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Next flush: 232","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263225708Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:05:17.263246249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"":"downstairs","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.26327726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263296749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"0","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.26331791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"1","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263338832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"client":"2","":"downstairs","session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263360294Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263380313Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"ba3a80c4-b64e-4e82-b8a9-41069a543d2d is now active with session: 18b4b962-2363-41aa-ad47-dac38616a44d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263402658Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263422056Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704,"session_id":"18b4b962-2363-41aa-ad47-dac38616a44d"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.263441835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} {"msg":"Activated sub_volume ba3a80c4-b64e-4e82-b8a9-41069a543d2d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:17.26345946Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 559 len 4096 data: 10 02/30 Read block 192 len 24576 03/30 Read block 21 len 12288 04/30 Flush 05/30 Write block 582 len 24576 data: 7 8 8 7 6 8 06/30 Write block 105 len 36864 data: 10 9 6 7 9 12 11 14 14 07/30 Read block 495 len 40960 08/30 Read block 293 len 20480 09/30 Read block 359 len 32768 10/30 Read block 475 len 12288 11/30 Read block 367 len 40960 12/30 Read block 304 len 32768 13/30 Write block 25 len 32768 data: 6 6 10 12 15 13 10 12 14/30 Read block 379 len 8192 15/30 Read block 113 len 32768 16/30 Read block 511 len 36864 17/30 Read block 416 len 16384 18/30 Read block 225 len 4096 19/30 Read block 69 len 12288 20/30 Read block 182 len 28672 21/30 Read block 488 len 16384 22/30 Read block 494 len 4096 23/30 Read block 27 len 32768 24/30 Write block 177 len 4096 data: 8 25/30 Write block 96 len 32768 data: 6 5 5 6 7 8 8 9 26/30 Read block 244 len 28672 27/30 Write block 241 len 12288 data: 13 15 16 28/30 Read block 407 len 40960 29/30 Write block 401 len 24576 data: 16 15 18 14 14 11 30/30 Write block 184 len 40960 data: 11 12 12 11 10 9 10 9 9 7 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Sent Done false 1001 Acked Read 6 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 6 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) None Some(JobId(1003)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:17.292806101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1704} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 229 229 229 F F F 1 020-039 126 126 126 229 229 229 T F T <--- 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 127 127 127 231 231 231 T F T <--- 5 100-119 126 126 126 230 230 230 T F T <--- 6 120-139 126 126 126 229 229 229 F F F 7 140-159 127 127 127 231 231 231 F F F 8 160-179 122 122 122 222 222 222 T F T <--- 9 180-199 124 124 124 224 224 224 T F T <--- 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 126 126 230 230 230 F F F 12 240-259 127 127 127 231 231 231 T F T <--- 13 260-279 125 125 125 228 228 228 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 127 127 127 231 231 231 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 124 224 224 224 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 127 127 127 231 231 231 F F F 20 400-419 127 127 127 231 231 231 T F T <--- 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 126 126 230 230 230 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 126 126 126 229 229 229 F F F 25 500-519 127 127 127 231 231 231 F F F 26 520-539 127 127 127 231 231 231 F F F 27 540-559 128 125 128 232 228 232 F F F <--- 28 560-579 127 127 127 231 231 231 F F F 29 580-599 124 124 124 225 225 225 T F T <--- Max gen: 128, Max flush: 232 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 229 229 229 F F F 1 020-039 129 129 129 233 233 233 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 120 120 217 217 217 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 126 126 126 229 229 229 F F F 7 140-159 127 127 127 231 231 231 F F F 8 160-179 129 129 129 233 233 233 F F F 9 180-199 129 129 129 233 233 233 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 126 126 230 230 230 F F F 12 240-259 129 129 129 233 233 233 F F F 13 260-279 125 125 125 228 228 228 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 127 127 127 231 231 231 F F F 16 320-339 119 119 119 216 216 216 F F F 17 340-359 124 124 124 224 224 224 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 127 127 127 231 231 231 F F F 20 400-419 129 129 129 233 233 233 F F F 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 126 126 230 230 230 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 126 126 126 229 229 229 F F F 25 500-519 127 127 127 231 231 231 F F F 26 520-539 127 127 127 231 231 231 F F F 27 540-559 128 128 128 232 232 232 F F F 28 560-579 127 127 127 231 231 231 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 129, Max flush: 233 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.826756949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:21.84613033Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.84622154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.863320702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.864016756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.86412545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.864150841Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.864190028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"Crucible f55d8b21-0048-48e3-bc49-3e23d5541599 has session id: 553cd8fb-aa80-4961-aa0c-99d038727e4a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.86536168Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Upstairs opts: Upstairs UUID: f55d8b21-0048-48e3-bc49-3e23d5541599, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.865424895Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Crucible stats registered with UUID: f55d8b21-0048-48e3-bc49-3e23d5541599","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.865457539Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:21.865484402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.865837559Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.865892588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.865924911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866185035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866217529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"f55d8b21-0048-48e3-bc49-3e23d5541599 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.86634469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.86637517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866403155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866431471Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866458905Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866487642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866609933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45080)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.866997017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46274)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.867096143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:33313)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.867132174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"io task","client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.868960993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.869018135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.869247899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.869997077Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870040723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[0]R flush_numbers[0..12]: [229, 233, 214, 217, 233, 233, 229, 231, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870159628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[0]R generation[0..12]: [126, 129, 117, 120, 129, 129, 126, 127, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870197342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870231009Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[1]R flush_numbers[0..12]: [229, 233, 214, 217, 233, 233, 229, 231, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870263263Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[1]R generation[0..12]: [126, 129, 117, 120, 129, 129, 126, 127, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870294875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870328141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[2]R flush_numbers[0..12]: [229, 233, 214, 217, 233, 233, 229, 231, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870359783Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[2]R generation[0..12]: [126, 129, 117, 120, 129, 129, 126, 127, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870391506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870424551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870451314Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870480251Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Next flush: 234","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870506853Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870543916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"":"downstairs","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870579406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870605388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"0","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870633383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"1","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870662941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"client":"2","":"downstairs","session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870691678Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.870718401Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"f55d8b21-0048-48e3-bc49-3e23d5541599 is now active with session: 553cd8fb-aa80-4961-aa0c-99d038727e4a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.87074848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.87077365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712,"session_id":"553cd8fb-aa80-4961-aa0c-99d038727e4a"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"crucible","level":30,"time":"2026-03-06T06:05:21.870810022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} {"msg":"Activated sub_volume f55d8b21-0048-48e3-bc49-3e23d5541599","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:21.87083447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 399 len 32768 data: 14 14 17 16 19 15 15 12 02/30 Write block 525 len 24576 data: 15 11 10 11 9 9 03/30 Write block 146 len 32768 data: 11 8 9 6 9 10 9 10 04/30 Write block 510 len 8192 data: 13 12 05/30 Write block 4 len 36864 data: 6 6 7 6 10 12 15 14 15 06/30 Read block 434 len 40960 07/30 Read block 470 len 24576 08/30 Read block 144 len 36864 09/30 Read block 427 len 40960 10/30 Read block 564 len 36864 11/30 Write block 64 len 32768 data: 8 9 11 13 16 13 11 9 12/30 Read block 39 len 8192 13/30 Write block 519 len 28672 data: 12 13 13 12 11 12 16 14/30 Write block 121 len 32768 data: 13 11 12 14 11 10 12 9 15/30 Flush 16/30 Write block 341 len 28672 data: 9 9 7 7 7 10 11 17/30 Read block 39 len 24576 18/30 Read block 57 len 28672 19/30 Read block 363 len 20480 20/30 Read block 546 len 40960 21/30 Read block 423 len 24576 22/30 Read block 107 len 8192 23/30 Write block 336 len 20480 data: 10 9 9 8 6 24/30 Write block 255 len 24576 data: 6 10 11 12 15 13 25/30 Write block 16 len 28672 data: 13 12 11 9 9 10 6 26/30 Write block 308 len 16384 data: 9 9 8 11 27/30 Write block 484 len 28672 data: 17 13 9 8 8 9 8 28/30 Write block 551 len 28672 data: 11 12 10 8 9 8 9 29/30 Read block 431 len 4096 30/30 Read block 380 len 24576 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 6 Sent Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:21.899109046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1712} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 130 130 229 234 234 F T T <--- 1 020-039 129 129 129 233 233 233 F T T <--- 2 040-059 117 117 117 214 214 214 F F F 3 060-079 120 130 130 217 234 234 F F F <--- 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 126 130 130 229 234 234 F F F <--- 7 140-159 127 130 130 231 234 234 F F F <--- 8 160-179 129 129 129 233 233 233 F F F 9 180-199 129 129 129 233 233 233 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 126 126 230 230 230 F F F 12 240-259 129 129 129 233 233 233 F T T <--- 13 260-279 125 125 125 228 228 228 F T T <--- 14 280-299 127 127 127 231 231 231 F F F 15 300-319 127 127 127 231 231 231 F T T <--- 16 320-339 119 119 119 216 216 216 F T T <--- 17 340-359 124 124 124 224 224 224 F T T <--- 18 360-379 126 126 126 229 229 229 F F F 19 380-399 127 130 130 231 234 234 T F F <--- 20 400-419 129 130 130 233 234 234 T F F <--- 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 126 126 230 230 230 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 126 126 126 229 229 229 F T T <--- 25 500-519 127 130 130 231 234 234 F F F <--- 26 520-539 127 130 130 231 234 234 F F F <--- 27 540-559 128 128 128 232 232 232 F T T <--- 28 560-579 127 127 127 231 231 231 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 130, Max flush: 234 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 235 235 235 F F F 1 020-039 131 131 131 235 235 235 F F F 2 040-059 117 117 117 214 214 214 F F F 3 060-079 130 130 130 234 234 234 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 130 130 130 234 234 234 F F F 7 140-159 130 130 130 234 234 234 F F F 8 160-179 129 129 129 233 233 233 F F F 9 180-199 129 129 129 233 233 233 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 126 126 126 230 230 230 F F F 12 240-259 131 131 131 235 235 235 F F F 13 260-279 131 131 131 235 235 235 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 131 131 131 235 235 235 F F F 16 320-339 131 131 131 235 235 235 F F F 17 340-359 131 131 131 235 235 235 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 130 130 130 234 234 234 F F F 20 400-419 130 130 130 234 234 234 F F F 21 420-439 125 125 125 228 228 228 F F F 22 440-459 126 126 126 230 230 230 F F F 23 460-479 123 123 123 223 223 223 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 130 130 130 234 234 234 F F F 26 520-539 130 130 130 234 234 234 F F F 27 540-559 131 131 131 235 235 235 F F F 28 560-579 127 127 127 231 231 231 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 131, Max flush: 235 Begin loop 66 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.475731953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.496134935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.496684891Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.496822873Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.496854866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.496898763Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"Crucible 7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9 has session id: 38f0d054-c9a0-467c-9eb6-d66b33777d26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.497699804Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Upstairs opts: Upstairs UUID: 7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.497756515Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Crucible stats registered with UUID: 7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.49778429Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:26.497808799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.498461547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.498570391Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.498599299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.498801167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.498834112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499033836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499065989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499092081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499118503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499144494Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499353547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499392564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"ds_connection connected from Ok(127.0.0.1:45895)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499465899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"ds_connection connected from Ok(127.0.0.1:62081)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499502882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"ds_connection connected from Ok(127.0.0.1:41376)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.499531298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"io task","client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.501197626Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.501335879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.501701791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.501896906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502143262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[0]R flush_numbers[0..12]: [235, 235, 214, 234, 233, 233, 234, 234, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.50264379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[0]R generation[0..12]: [131, 131, 117, 130, 129, 129, 130, 130, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502678518Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502710131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[1]R flush_numbers[0..12]: [235, 235, 214, 234, 233, 233, 234, 234, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.50274Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[1]R generation[0..12]: [131, 131, 117, 130, 129, 129, 130, 130, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502768917Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502805589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[2]R flush_numbers[0..12]: [235, 235, 214, 234, 233, 233, 234, 234, 233, 233, 219, 230]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502835087Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[2]R generation[0..12]: [131, 131, 117, 130, 129, 129, 130, 130, 129, 129, 121, 126]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502864365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502895266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502920496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.502946667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Next flush: 236","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:05:26.502971496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503002458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"":"downstairs","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.50302913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503052546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"0","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503078267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"1","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503103948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"client":"2","":"downstairs","session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.50313024Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503154828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9 is now active with session: 38f0d054-c9a0-467c-9eb6-d66b33777d26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503182663Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.503206029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720,"session_id":"38f0d054-c9a0-467c-9eb6-d66b33777d26"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:05:26.503234806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} {"msg":"Activated sub_volume 7126b0a7-3cdf-4816-bb1c-8c6b4b10dec9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:26.50326226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 465 len 20480 data: 11 9 7 8 6 02/30 Write block 30 len 4096 data: 14 03/30 Write block 157 len 36864 data: 10 7 7 8 9 13 12 11 10 04/30 Flush 05/30 Write block 452 len 24576 data: 10 10 9 9 8 11 06/30 Read block 288 len 40960 07/30 Read block 291 len 24576 08/30 Write block 541 len 8192 data: 12 13 09/30 Write block 1 len 24576 data: 3 5 6 7 7 8 10/30 Read block 286 len 4096 11/30 Read block 19 len 16384 12/30 Write block 167 len 24576 data: 11 10 10 11 12 9 13/30 Read block 380 len 4096 14/30 Write block 419 len 24576 data: 6 9 10 10 10 11 15/30 Write block 315 len 24576 data: 13 11 9 10 9 9 16/30 Read block 210 len 32768 17/30 Write block 137 len 8192 data: 7 5 18/30 Write block 156 len 40960 data: 10 11 8 8 9 10 14 13 12 11 19/30 Write block 550 len 12288 data: 12 12 13 20/30 Write block 395 len 12288 data: 5 5 7 21/30 Write block 239 len 12288 data: 12 13 14 22/30 Write block 560 len 8192 data: 7 6 23/30 Flush 24/30 Write block 572 len 32768 data: 9 9 9 8 10 9 7 10 25/30 Read block 524 len 16384 26/30 Write block 56 len 16384 data: 8 8 8 8 27/30 Read block 575 len 20480 28/30 Write block 463 len 28672 data: 10 9 12 10 8 9 7 29/30 Write block 179 len 8192 data: 7 8 30/30 Write block 313 len 12288 data: 13 12 14 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 6 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Write 3 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 28 3 34 Done 27 2 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:26.527835592Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1720} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 131 132 237 235 237 F F F <--- 1 020-039 132 131 132 236 235 236 F T F <--- 2 040-059 117 117 117 214 214 214 T F T <--- 3 060-079 130 130 130 234 234 234 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 130 132 237 234 237 F F F <--- 7 140-159 132 130 132 237 234 237 F F F <--- 8 160-179 132 129 132 237 233 237 T F T <--- 9 180-199 129 129 129 233 233 233 T F T <--- 10 200-219 121 121 121 219 219 219 F F F 11 220-239 132 126 132 237 230 237 F F F <--- 12 240-259 132 131 132 237 235 237 F F F <--- 13 260-279 131 131 131 235 235 235 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 132 131 132 237 235 237 T F T <--- 16 320-339 132 131 132 237 235 237 F F F <--- 17 340-359 131 131 131 235 235 235 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 132 130 132 237 234 237 F F F <--- 20 400-419 132 130 132 237 234 237 F F F <--- 21 420-439 132 125 132 237 228 237 F F F <--- 22 440-459 132 126 132 237 230 237 F F F <--- 23 460-479 132 123 132 236 223 236 T T T <--- 24 480-499 131 131 131 235 235 235 F F F 25 500-519 130 130 130 234 234 234 F F F 26 520-539 130 130 130 234 234 234 F F F 27 540-559 132 131 132 237 235 237 F F F <--- 28 560-579 132 127 132 237 231 237 T F T <--- 29 580-599 129 129 129 233 233 233 F F F Max gen: 132, Max flush: 237 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 130 130 130 234 234 234 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 132 132 132 237 237 237 F F F 8 160-179 133 133 133 238 238 238 F F F 9 180-199 133 133 133 238 238 238 F F F 10 200-219 121 121 121 219 219 219 F F F 11 220-239 132 132 132 237 237 237 F F F 12 240-259 132 132 132 237 237 237 F F F 13 260-279 131 131 131 235 235 235 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 133 133 133 238 238 238 F F F 16 320-339 132 132 132 237 237 237 F F F 17 340-359 131 131 131 235 235 235 F F F 18 360-379 126 126 126 229 229 229 F F F 19 380-399 132 132 132 237 237 237 F F F 20 400-419 132 132 132 237 237 237 F F F 21 420-439 132 132 132 237 237 237 F F F 22 440-459 132 132 132 237 237 237 F F F 23 460-479 133 133 133 238 238 238 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 130 130 130 234 234 234 F F F 26 520-539 130 130 130 234 234 234 F F F 27 540-559 132 132 132 237 237 237 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 133, Max flush: 238 Begin loop 67 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.127361775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:31.147949302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.148051734Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.166674343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.167250971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.167382481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.16741228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.167453791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"Crucible 3eaa3756-d8f1-4b6d-ad0b-77714d565a7f has session id: 9dee86ba-a0de-47e9-84fc-0c1a634fb398","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.168501529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Upstairs opts: Upstairs UUID: 3eaa3756-d8f1-4b6d-ad0b-77714d565a7f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.168579252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Crucible stats registered with UUID: 3eaa3756-d8f1-4b6d-ad0b-77714d565a7f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.168614041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:31.168643108Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.168957779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169017056Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169183795Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169384712Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169419561Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"3eaa3756-d8f1-4b6d-ad0b-77714d565a7f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169555399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169587171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169618143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169649023Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169679444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169711006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169742468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"ds_connection connected from Ok(127.0.0.1:54714)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169928104Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"ds_connection connected from Ok(127.0.0.1:51609)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169964456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"ds_connection connected from Ok(127.0.0.1:35138)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.169997641Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"io task","client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.171599212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.171750811Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.172123817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.17232791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.172512324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 238, 238, 219, 237]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.172891081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 133, 133, 121, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.172930459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.172966801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 238, 238, 219, 237]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.17300191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 133, 133, 121, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173036779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173073211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 238, 238, 219, 237]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173107939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 133, 133, 121, 132]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173142367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173178428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173208407Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173244429Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Next flush: 239","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.17328589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173315268Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"":"downstairs","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173346229Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173374305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"0","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173404765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"1","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173435145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"client":"2","":"downstairs","session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173466346Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173495324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"3eaa3756-d8f1-4b6d-ad0b-77714d565a7f is now active with session: 9dee86ba-a0de-47e9-84fc-0c1a634fb398","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173528168Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-06T06:05:31.173555653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728,"session_id":"9dee86ba-a0de-47e9-84fc-0c1a634fb398"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173592816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} {"msg":"Activated sub_volume 3eaa3756-d8f1-4b6d-ad0b-77714d565a7f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:31.173618497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 272 len 28672 data: 10 13 10 14 16 17 18 02/30 Write block 410 len 20480 data: 10 11 11 8 10 03/30 Write block 542 len 12288 data: 14 11 10 04/30 Write block 167 len 4096 data: 12 05/30 Read block 241 len 36864 06/30 Read block 496 len 32768 07/30 Write block 303 len 40960 data: 12 6 7 8 8 10 10 9 12 12 08/30 Flush 09/30 Write block 415 len 32768 data: 8 8 7 6 7 10 11 11 10/30 Read block 77 len 36864 11/30 Read block 139 len 4096 12/30 Write block 419 len 36864 data: 8 11 12 12 11 12 7 5 8 13/30 Read block 241 len 12288 14/30 Read block 555 len 12288 15/30 Read block 51 len 16384 16/30 Flush 17/30 Write block 529 len 24576 data: 10 10 10 10 11 8 18/30 Write block 360 len 36864 data: 7 8 9 9 12 12 11 10 9 19/30 Write block 223 len 4096 data: 10 20/30 Write block 176 len 24576 data: 7 9 8 8 9 9 21/30 Read block 45 len 32768 22/30 Read block 213 len 36864 23/30 Write block 398 len 36864 data: 9 15 15 18 17 20 16 16 13 24/30 Write block 217 len 12288 data: 8 9 12 25/30 Read block 55 len 40960 26/30 Read block 378 len 28672 27/30 Read block 542 len 8192 28/30 Read block 208 len 20480 29/30 Read block 594 len 16384 30/30 Write block 242 len 24576 data: 16 17 14 14 12 11 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:31.203950683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1728} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 130 130 130 234 234 234 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 132 132 132 237 237 237 F F F 8 160-179 133 134 134 238 239 239 F T T <--- 9 180-199 133 133 133 238 238 238 F T T <--- 10 200-219 121 121 121 219 219 219 F T T <--- 11 220-239 132 132 132 237 237 237 F T T <--- 12 240-259 132 132 132 237 237 237 F T T <--- 13 260-279 131 134 134 235 239 239 F F F <--- 14 280-299 127 127 127 231 231 231 F F F 15 300-319 133 134 134 238 239 239 F F F <--- 16 320-339 132 132 132 237 237 237 F F F 17 340-359 131 131 131 235 235 235 F F F 18 360-379 126 126 126 229 229 229 F T T <--- 19 380-399 132 132 132 237 237 237 F T T <--- 20 400-419 132 134 134 237 240 240 F T T <--- 21 420-439 132 134 134 237 240 240 F F F <--- 22 440-459 132 132 132 237 237 237 F F F 23 460-479 133 133 133 238 238 238 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 130 130 130 234 234 234 F F F 26 520-539 130 130 130 234 234 234 F T T <--- 27 540-559 132 134 134 237 239 239 F F F <--- 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 134, Max flush: 240 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 130 130 130 234 234 234 F F F 4 080-099 129 129 129 233 233 233 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 132 132 132 237 237 237 F F F 8 160-179 135 135 135 241 241 241 F F F 9 180-199 135 135 135 241 241 241 F F F 10 200-219 135 135 135 241 241 241 F F F 11 220-239 135 135 135 241 241 241 F F F 12 240-259 135 135 135 241 241 241 F F F 13 260-279 134 134 134 239 239 239 F F F 14 280-299 127 127 127 231 231 231 F F F 15 300-319 134 134 134 239 239 239 F F F 16 320-339 132 132 132 237 237 237 F F F 17 340-359 131 131 131 235 235 235 F F F 18 360-379 135 135 135 241 241 241 F F F 19 380-399 135 135 135 241 241 241 F F F 20 400-419 135 135 135 241 241 241 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 132 132 132 237 237 237 F F F 23 460-479 133 133 133 238 238 238 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 130 130 130 234 234 234 F F F 26 520-539 135 135 135 241 241 241 F F F 27 540-559 134 134 134 239 239 239 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 135, Max flush: 241 Begin loop 68 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.778916128Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.800994527Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.801498942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.801616444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.801663507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.801702163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"Crucible 2f32843a-e343-41f7-b47c-b74e872b5253 has session id: 1563fc3e-a8c6-45df-94a8-f9c287be405b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.802900147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Upstairs opts: Upstairs UUID: 2f32843a-e343-41f7-b47c-b74e872b5253, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.80296219Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Crucible stats registered with UUID: 2f32843a-e343-41f7-b47c-b74e872b5253","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.80299258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:35.803019012Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.803558957Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.803616551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.803646109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.803946171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.803983886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"2f32843a-e343-41f7-b47c-b74e872b5253 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804134563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804166075Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804196866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804233438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804270862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804302885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804334608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"ds_connection connected from Ok(127.0.0.1:34058)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804724998Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37225)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.804975022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"ds_connection connected from Ok(127.0.0.1:35359)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.805015882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"io task","client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.806420444Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.806472056Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.807300521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.807339357Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.807372262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 241, 241, 241, 241]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808300865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 135, 135, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808339531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808373187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 241, 241, 241, 241]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808412014Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 135, 135, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808444067Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808476862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 234, 233, 233, 237, 237, 241, 241, 241, 241]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808508575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 130, 129, 129, 132, 132, 135, 135, 135, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808540107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808572932Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808600055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808628281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Next flush: 242","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808654933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808682418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"":"downstairs","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808710503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808736334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"0","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808764209Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"1","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808791984Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"client":"2","":"downstairs","session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.80882049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808850349Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"2f32843a-e343-41f7-b47c-b74e872b5253 is now active with session: 1563fc3e-a8c6-45df-94a8-f9c287be405b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808880509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"Wait for a query_work_queue command to finish before sending IO 1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808912612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736,"session_id":"1563fc3e-a8c6-45df-94a8-f9c287be405b"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808939154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} {"msg":"Activated sub_volume 2f32843a-e343-41f7-b47c-b74e872b5253","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:35.808961989Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 169 len 24576 data: 11 12 13 10 7 8 02/30 Write block 226 len 4096 data: 12 03/30 Write block 451 len 24576 data: 11 11 11 10 10 9 04/30 Write block 291 len 32768 data: 12 11 9 9 13 14 16 13 05/30 Read block 139 len 32768 06/30 Write block 330 len 20480 data: 14 10 8 9 9 07/30 Write block 452 len 28672 data: 12 12 11 11 10 12 13 08/30 Read block 122 len 36864 09/30 Write block 450 len 20480 data: 10 12 13 13 12 10/30 Read block 516 len 40960 11/30 Write block 320 len 40960 data: 10 9 8 6 5 6 6 11 12 13 12/30 Write block 167 len 36864 data: 13 11 12 13 14 11 8 9 7 13/30 Flush 14/30 Write block 275 len 36864 data: 15 17 18 19 16 16 16 16 15 15/30 Read block 49 len 36864 16/30 Write block 505 len 16384 data: 9 9 12 13 17/30 Read block 338 len 24576 18/30 Read block 12 len 28672 19/30 Flush 20/30 Write block 470 len 20480 data: 9 11 10 11 11 21/30 Flush 22/30 Write block 231 len 8192 data: 9 9 23/30 Write block 147 len 28672 data: 9 10 7 10 11 10 11 24/30 Read block 320 len 12288 25/30 Write block 197 len 40960 data: 9 10 8 8 7 7 8 8 6 9 26/30 Write block 440 len 24576 data: 10 14 13 13 10 13 27/30 Write block 73 len 32768 data: 11 12 14 15 16 13 12 13 28/30 Read block 320 len 12288 29/30 Write block 346 len 40960 data: 11 12 11 10 10 9 8 9 7 9 30/30 Read block 413 len 28672 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 10 Done Done Sent false 1029 Acked Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 26 27 Done 29 30 4 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:35.838527653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1736} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 130 130 130 234 234 234 T T F <--- 4 080-099 129 129 129 233 233 233 T T F <--- 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 132 132 132 237 237 237 T T F <--- 8 160-179 136 136 135 242 242 241 F F T <--- 9 180-199 135 135 135 241 241 241 T T F <--- 10 200-219 135 135 135 241 241 241 T T F <--- 11 220-239 136 136 135 242 242 241 T T T <--- 12 240-259 135 135 135 241 241 241 F F F 13 260-279 136 136 134 243 243 239 F F F <--- 14 280-299 136 136 127 243 243 231 F F T <--- 15 300-319 134 134 134 239 239 239 F F F 16 320-339 136 136 132 242 242 237 F F F <--- 17 340-359 131 131 131 235 235 235 T T F <--- 18 360-379 135 135 135 241 241 241 F F F 19 380-399 135 135 135 241 241 241 F F F 20 400-419 135 135 135 241 241 241 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 136 136 132 242 242 237 T T T <--- 23 460-479 136 136 133 244 244 238 F F F <--- 24 480-499 131 131 131 235 235 235 F F F 25 500-519 136 136 130 243 243 234 F F F <--- 26 520-539 135 135 135 241 241 241 F F F 27 540-559 134 134 134 239 239 239 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 136, Max flush: 244 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 137 137 137 245 245 245 F F F 4 080-099 137 137 137 245 245 245 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 136 136 136 242 242 242 F F F 9 180-199 137 137 137 245 245 245 F F F 10 200-219 137 137 137 245 245 245 F F F 11 220-239 137 137 137 245 245 245 F F F 12 240-259 135 135 135 241 241 241 F F F 13 260-279 136 136 136 243 243 243 F F F 14 280-299 136 136 136 243 243 243 F F F 15 300-319 134 134 134 239 239 239 F F F 16 320-339 136 136 136 242 242 242 F F F 17 340-359 137 137 137 245 245 245 F F F 18 360-379 135 135 135 241 241 241 F F F 19 380-399 135 135 135 241 241 241 F F F 20 400-419 135 135 135 241 241 241 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 136 136 136 244 244 244 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 136 136 136 243 243 243 F F F 26 520-539 135 135 135 241 241 241 F F F 27 540-559 134 134 134 239 239 239 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 129 129 233 233 233 F F F Max gen: 137, Max flush: 245 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.40636387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:40.426454216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.4265525Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.443565112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.444004639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.444074928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.444097683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.444129796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"Crucible 7b715e22-e1d7-46fb-b61f-26feecfb3301 has session id: 788fcd2b-409e-42d2-aba6-a62aa9aba8a6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.445009322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Upstairs opts: Upstairs UUID: 7b715e22-e1d7-46fb-b61f-26feecfb3301, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.445327329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Crucible stats registered with UUID: 7b715e22-e1d7-46fb-b61f-26feecfb3301","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.445363591Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:40.445386827Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.445742509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.44579396Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.445820322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446169251Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.4461995Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"7b715e22-e1d7-46fb-b61f-26feecfb3301 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446413222Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446440496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446464864Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446488681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446517097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.44659981Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.44662999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:35419)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446907958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:47338)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.446996533Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:48149)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.447027534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"io task","client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.448973905Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449029866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449198879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449728404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449769245Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 245, 245, 233, 237, 245, 242, 245, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.44993402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 137, 137, 129, 132, 137, 136, 137, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449967897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.449994449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 245, 245, 233, 237, 245, 242, 245, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450019308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 137, 137, 129, 132, 137, 136, 137, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450043837Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450068505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 245, 245, 233, 237, 245, 242, 245, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450092342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 137, 137, 129, 132, 137, 136, 137, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450125237Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450150938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450171749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450193342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Next flush: 246","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450213672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450233972Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"":"downstairs","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450255594Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450275163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"0","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450296996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"1","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450318338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"client":"2","":"downstairs","session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Wait for a query_work_queue command to finish before sending IO All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450348658Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450370451Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"7b715e22-e1d7-46fb-b61f-26feecfb3301 is now active with session: 788fcd2b-409e-42d2-aba6-a62aa9aba8a6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450393326Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450412574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744,"session_id":"788fcd2b-409e-42d2-aba6-a62aa9aba8a6"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.450433035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} {"msg":"Activated sub_volume 7b715e22-e1d7-46fb-b61f-26feecfb3301","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:40.45045095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 90 len 32768 data: 15 13 14 13 12 10 7 6 02/30 Write block 188 len 40960 data: 11 10 11 10 10 8 10 10 11 10 03/30 Read block 418 len 12288 04/30 Flush 05/30 Write block 381 len 12288 data: 23 19 18 06/30 Write block 339 len 24576 data: 9 7 10 10 8 8 07/30 Read block 300 len 4096 08/30 Flush 09/30 Flush 10/30 Write block 466 len 32768 data: 11 9 10 8 10 12 11 12 11/30 Flush 12/30 Write block 583 len 20480 data: 9 9 8 7 9 13/30 Flush 14/30 Write block 280 len 36864 data: 17 17 17 16 15 15 14 10 11 15/30 Read block 174 len 4096 16/30 Write block 240 len 32768 data: 14 15 17 18 15 15 13 12 17/30 Read block 291 len 8192 18/30 Read block 469 len 16384 19/30 Write block 67 len 20480 data: 14 17 14 12 10 20/30 Write block 410 len 12288 data: 11 12 12 21/30 Write block 468 len 32768 data: 11 9 11 13 12 13 12 13 22/30 Read block 128 len 16384 23/30 Read block 444 len 40960 24/30 Read block 469 len 20480 25/30 Read block 9 len 20480 26/30 Read block 102 len 16384 27/30 Read block 232 len 24576 28/30 Read block 141 len 28672 29/30 Read block 368 len 8192 30/30 Read block 241 len 4096 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:40.479810938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1744} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 137 137 137 245 245 245 F T T <--- 4 080-099 137 138 138 245 246 246 T F F <--- 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 136 136 136 242 242 242 F F F 9 180-199 137 138 138 245 246 246 F F F <--- 10 200-219 137 137 137 245 245 245 F F F 11 220-239 137 137 137 245 245 245 F F F 12 240-259 135 135 135 241 241 241 F T T <--- 13 260-279 136 136 136 243 243 243 F F F 14 280-299 136 136 136 243 243 243 F T T <--- 15 300-319 134 134 134 239 239 239 F F F 16 320-339 136 138 138 242 247 247 F F F <--- 17 340-359 137 138 138 245 247 247 F F F <--- 18 360-379 135 135 135 241 241 241 F F F 19 380-399 135 138 138 241 247 247 F F F <--- 20 400-419 135 135 135 241 241 241 F T T <--- 21 420-439 134 134 134 240 240 240 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 136 138 138 244 249 249 F T T <--- 24 480-499 131 131 131 235 235 235 F F F 25 500-519 136 136 136 243 243 243 F F F 26 520-539 135 135 135 241 241 241 F F F 27 540-559 134 134 134 239 239 239 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 129 138 138 233 250 250 F F F <--- Max gen: 138, Max flush: 250 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 139 139 139 251 251 251 F F F 4 080-099 138 138 138 246 246 246 F F F 5 100-119 129 129 129 233 233 233 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 136 136 136 242 242 242 F F F 9 180-199 138 138 138 246 246 246 F F F 10 200-219 137 137 137 245 245 245 F F F 11 220-239 137 137 137 245 245 245 F F F 12 240-259 139 139 139 251 251 251 F F F 13 260-279 136 136 136 243 243 243 F F F 14 280-299 139 139 139 251 251 251 F F F 15 300-319 134 134 134 239 239 239 F F F 16 320-339 138 138 138 247 247 247 F F F 17 340-359 138 138 138 247 247 247 F F F 18 360-379 135 135 135 241 241 241 F F F 19 380-399 138 138 138 247 247 247 F F F 20 400-419 139 139 139 251 251 251 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 136 136 136 243 243 243 F F F 26 520-539 135 135 135 241 241 241 F F F 27 540-559 134 134 134 239 239 239 F F F 28 560-579 133 133 133 238 238 238 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 139, Max flush: 251 Begin loop 70 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.040757788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.058999445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.059799695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.059881206Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.059908529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.059945573Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"Crucible 5013e6ca-66f4-41b0-80cd-03231a812494 has session id: 9443337d-3d1e-4369-bd21-6f86b64417dc","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.060905036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Upstairs opts: Upstairs UUID: 5013e6ca-66f4-41b0-80cd-03231a812494, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061103367Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Crucible stats registered with UUID: 5013e6ca-66f4-41b0-80cd-03231a812494","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061156672Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:45.06119609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061679424Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061789551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061826354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.061984917Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062145935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"5013e6ca-66f4-41b0-80cd-03231a812494 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062182136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.06220923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062238317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062266903Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062294688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062323816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:63992)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.0627264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062804103Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:57854)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062836277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:35844)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.062866947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"io task","client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.064315386Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.064370555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.065142278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.065269288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.066607509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 251, 246, 233, 237, 245, 242, 246, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067640578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 139, 138, 129, 132, 137, 136, 138, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067724654Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067780985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 251, 246, 233, 237, 245, 242, 246, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067829711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 139, 138, 129, 132, 137, 136, 138, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067877275Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067926742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 251, 246, 233, 237, 245, 242, 246, 245, 245]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.067974537Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 139, 138, 129, 132, 137, 136, 138, 137, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.06802174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068069244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068096267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068124653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Next flush: 252","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068151055Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.0681792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"":"downstairs","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068207456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"Wait for a query_work_queue command to finish before sending IO session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068251203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"0","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068282935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"1","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068311472Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"client":"2","":"downstairs","session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068340308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068367201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"5013e6ca-66f4-41b0-80cd-03231a812494 is now active with session: 9443337d-3d1e-4369-bd21-6f86b64417dc","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068397381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068422691Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752,"session_id":"9443337d-3d1e-4369-bd21-6f86b64417dc"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068449484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} {"msg":"Activated sub_volume 5013e6ca-66f4-41b0-80cd-03231a812494","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:45.068472579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 374 len 28672 data: 11 12 11 15 18 22 21 02/30 Write block 296 len 12288 data: 15 17 14 03/30 Read block 291 len 20480 04/30 Write block 561 len 8192 data: 7 8 05/30 Read block 4 len 4096 06/30 Flush 07/30 Write block 216 len 24576 data: 5 9 10 13 12 11 08/30 Write block 505 len 28672 data: 10 10 13 14 13 14 13 09/30 Read block 308 len 20480 10/30 Read block 587 len 32768 11/30 Read block 142 len 24576 12/30 Write block 80 len 28672 data: 14 17 20 18 15 16 15 13/30 Read block 386 len 40960 14/30 Read block 509 len 32768 15/30 Read block 106 len 16384 16/30 Write block 116 len 12288 data: 15 13 14 17/30 Write block 352 len 40960 data: 9 10 8 10 11 7 6 8 8 9 18/30 Write block 538 len 40960 data: 13 14 11 13 15 12 11 9 9 10 19/30 Write block 556 len 12288 data: 9 10 10 20/30 Flush 21/30 Write block 175 len 28672 data: 8 8 10 9 9 10 10 22/30 Read block 435 len 28672 23/30 Write block 248 len 32768 data: 12 9 7 6 5 7 7 7 24/30 Write block 320 len 24576 data: 11 10 9 7 6 7 25/30 Write block 334 len 40960 data: 10 9 11 10 10 10 8 11 11 9 26/30 Write block 67 len 4096 data: 15 27/30 Write block 74 len 32768 data: 13 15 16 17 14 13 15 18 28/30 Read block 401 len 24576 29/30 Read block 133 len 32768 30/30 Read block 368 len 24576 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Sent false 1001 Acked Write 3 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Write 2 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Write 7 Done Done Sent false 1008 Acked Read 5 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:45.099888054Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1752} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 139 139 139 251 251 251 T T F <--- 4 080-099 140 140 138 253 253 246 T T F <--- 5 100-119 140 140 129 253 253 233 F F F <--- 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 136 136 136 242 242 242 T T F <--- 9 180-199 138 138 138 246 246 246 T T F <--- 10 200-219 140 140 137 253 253 245 F F F <--- 11 220-239 140 140 137 253 253 245 F F F <--- 12 240-259 139 139 139 251 251 251 T T F <--- 13 260-279 136 136 136 243 243 243 F F F 14 280-299 140 140 139 252 252 251 F F F <--- 15 300-319 134 134 134 239 239 239 F F F 16 320-339 138 138 138 247 247 247 T T F <--- 17 340-359 140 140 138 253 253 247 T T F <--- 18 360-379 140 140 135 253 253 241 F F F <--- 19 380-399 140 140 138 252 252 247 F F F <--- 20 400-419 139 139 139 251 251 251 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 140 140 136 253 253 243 F F F <--- 26 520-539 140 140 135 253 253 241 F F F <--- 27 540-559 140 140 134 253 253 239 F F F <--- 28 560-579 140 140 133 252 252 238 F F F <--- 29 580-599 138 138 138 250 250 250 F F F Max gen: 140, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 140 140 140 253 253 253 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 141 141 141 254 254 254 F F F 9 180-199 141 141 141 254 254 254 F F F 10 200-219 140 140 140 253 253 253 F F F 11 220-239 140 140 140 253 253 253 F F F 12 240-259 141 141 141 254 254 254 F F F 13 260-279 136 136 136 243 243 243 F F F 14 280-299 140 140 140 252 252 252 F F F 15 300-319 134 134 134 239 239 239 F F F 16 320-339 141 141 141 254 254 254 F F F 17 340-359 141 141 141 254 254 254 F F F 18 360-379 140 140 140 253 253 253 F F F 19 380-399 140 140 140 252 252 252 F F F 20 400-419 139 139 139 251 251 251 F F F 21 420-439 134 134 134 240 240 240 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 140 140 140 253 253 253 F F F 26 520-539 140 140 140 253 253 253 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 140 140 140 252 252 252 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 141, Max flush: 254 Begin loop 71 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.700888095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.722665119Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.723725271Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.723829186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.723868985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.723922611Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"Crucible 03967d3c-7f68-4d0d-b134-3bc2987b6c16 has session id: 48cbfa74-4cdf-4f19-8e97-9740f46e8d66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.724936291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Upstairs opts: Upstairs UUID: 03967d3c-7f68-4d0d-b134-3bc2987b6c16, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.725030006Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Crucible stats registered with UUID: 03967d3c-7f68-4d0d-b134-3bc2987b6c16","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.725075686Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:49.725112719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.725534922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.725623607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.725649458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726052844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726087603Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"03967d3c-7f68-4d0d-b134-3bc2987b6c16 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726195325Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726221347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726244002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726266386Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.72628858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726311445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726404338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"ds_connection connected from Ok(127.0.0.1:54852)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726590616Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"ds_connection connected from Ok(127.0.0.1:40875)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726630244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"ds_connection connected from Ok(127.0.0.1:42257)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.726654321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"io task","client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.728997876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.729042334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.729527241Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.72999323Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730563906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 253, 237, 245, 254, 254, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730638744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 141, 141, 140, 132, 137, 141, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730673994Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730707991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 253, 237, 245, 254, 254, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730740114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 141, 141, 140, 132, 137, 141, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730771817Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730804862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 253, 237, 245, 254, 254, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730836644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 141, 141, 140, 132, 137, 141, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730872666Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730906001Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730933215Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.730961701Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:05:49.730988264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731049565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"":"downstairs","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731078702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731103992Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"0","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731132408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"1","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731160363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"client":"2","":"downstairs","session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731188809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731215242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"03967d3c-7f68-4d0d-b134-3bc2987b6c16 is now active with session: 48cbfa74-4cdf-4f19-8e97-9740f46e8d66","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731245211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.73127039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760,"session_id":"48cbfa74-4cdf-4f19-8e97-9740f46e8d66"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731297023Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} {"msg":"Activated sub_volume 03967d3c-7f68-4d0d-b134-3bc2987b6c16","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:49.731324136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 315 len 8192 data: 15 12 02/30 Read block 192 len 12288 03/30 Read block 337 len 40960 04/30 Write block 289 len 4096 data: 14 05/30 Flush 06/30 Write block 100 len 24576 data: 8 9 9 10 10 11 07/30 Read block 569 len 16384 08/30 Read block 267 len 8192 09/30 Read block 450 len 28672 10/30 Write block 412 len 28672 data: 13 9 11 9 9 8 7 11/30 Write block 563 len 16384 data: 9 9 11 11 12/30 Write block 279 len 20480 data: 17 18 18 18 17 13/30 Write block 323 len 40960 data: 8 7 8 7 12 13 14 15 11 9 14/30 Write block 391 len 12288 data: 5 5 6 15/30 Flush 16/30 Write block 431 len 32768 data: 10 7 6 10 12 13 14 14 17/30 Read block 241 len 8192 18/30 Write block 350 len 32768 data: 11 10 10 11 9 11 12 8 19/30 Write block 246 len 40960 data: 14 13 13 10 8 7 6 8 8 8 20/30 Write block 400 len 40960 data: 16 19 18 21 17 17 14 8 10 9 21/30 Write block 366 len 8192 data: 12 11 22/30 Read block 67 len 8192 23/30 Write block 283 len 8192 data: 18 16 24/30 Write block 294 len 36864 data: 10 14 16 18 15 12 12 9 13 25/30 Flush 26/30 Write block 317 len 40960 data: 10 11 10 12 11 10 9 8 9 8 27/30 Read block 260 len 16384 28/30 Read block 180 len 16384 29/30 Read block 71 len 36864 30/30 Write block 180 len 12288 data: 11 11 11 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:49.760353241Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1760} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 140 142 256 253 256 F F F <--- 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 141 141 141 254 254 254 F F F 9 180-199 141 141 141 254 254 254 T F T <--- 10 200-219 140 140 140 253 253 253 F F F 11 220-239 140 140 140 253 253 253 F F F 12 240-259 142 141 142 257 254 257 F F F <--- 13 260-279 142 136 142 256 243 256 F F F <--- 14 280-299 142 140 142 257 252 257 F F F <--- 15 300-319 142 134 142 257 239 257 T T T <--- 16 320-339 142 141 142 256 254 256 T F T <--- 17 340-359 142 141 142 257 254 257 F F F <--- 18 360-379 142 140 142 257 253 257 F F F <--- 19 380-399 142 140 142 256 252 256 F F F <--- 20 400-419 142 139 142 257 251 257 F F F <--- 21 420-439 142 134 142 257 240 257 F F F <--- 22 440-459 137 137 137 245 245 245 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 140 140 140 253 253 253 F F F 26 520-539 140 140 140 253 253 253 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 142 140 142 256 252 256 F F F <--- 29 580-599 138 138 138 250 250 250 F F F Max gen: 142, Max flush: 257 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F F F 8 160-179 141 141 141 254 254 254 F F F 9 180-199 143 143 143 258 258 258 F F F 10 200-219 140 140 140 253 253 253 F F F 11 220-239 140 140 140 253 253 253 F F F 12 240-259 142 142 142 257 257 257 F F F 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 143 143 143 258 258 258 F F F 16 320-339 143 143 143 258 258 258 F F F 17 340-359 142 142 142 257 257 257 F F F 18 360-379 142 142 142 257 257 257 F F F 19 380-399 142 142 142 256 256 256 F F F 20 400-419 142 142 142 257 257 257 F F F 21 420-439 142 142 142 257 257 257 F F F 22 440-459 137 137 137 245 245 245 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F F F 25 500-519 140 140 140 253 253 253 F F F 26 520-539 140 140 140 253 253 253 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 142 142 142 256 256 256 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 143, Max flush: 258 Begin loop 72 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.298039662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:54.316559238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.316739083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.333506331Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.333958092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.334029162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.334053761Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.334087838Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} {"msg":"Crucible 36d3516c-9f17-4a03-9d16-67073fdd49eb has session id: 36f50343-7a1c-432d-990a-bac5d39e3fd4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335156928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Upstairs opts: Upstairs UUID: 36d3516c-9f17-4a03-9d16-67073fdd49eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335236605Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Crucible stats registered with UUID: 36d3516c-9f17-4a03-9d16-67073fdd49eb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335275101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:54.335301503Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335622447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335800057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.335839325Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336358399Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336394731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} {"msg":"36d3516c-9f17-4a03-9d16-67073fdd49eb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336497974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336530609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336559085Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336783778Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336836041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.33686566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336894948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"ds_connection connected from Ok(127.0.0.1:45563)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336925057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"ds_connection connected from Ok(127.0.0.1:49334)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.336955457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"ds_connection connected from Ok(127.0.0.1:60858)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.337161564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"io task","client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.338757784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.338809856Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.339568844Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.339610987Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.339837064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[0]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 256, 237, 245, 254, 258, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340672101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[0]R generation[0..12]: [132, 132, 133, 141, 141, 142, 132, 137, 141, 143, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.34071198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340745807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[1]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 256, 237, 245, 254, 258, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.34077776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[1]R generation[0..12]: [132, 132, 133, 141, 141, 142, 132, 137, 141, 143, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340809252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340842397Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[2]R flush_numbers[0..12]: [237, 236, 238, 254, 254, 256, 237, 245, 254, 258, 253, 253]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.34087414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[2]R generation[0..12]: [132, 132, 133, 141, 141, 142, 132, 137, 141, 143, 140, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340905993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340939309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.340966332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341019056Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Next flush: 259","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341047162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341080748Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"":"downstairs","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341109434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341135035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"0","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.34116311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"1","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341190795Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"client":"2","":"downstairs","session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341219101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341245653Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"36d3516c-9f17-4a03-9d16-67073fdd49eb is now active with session: 36f50343-7a1c-432d-990a-bac5d39e3fd4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341275873Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341301273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768,"session_id":"36f50343-7a1c-432d-990a-bac5d39e3fd4"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341327845Z","hostname":"bmat-EVT22200007-0000ec3a","Wait for a query_work_queue command to finish before sending IO pid":1768} {"msg":"Activated sub_volume 36d3516c-9f17-4a03-9d16-67073fdd49eb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:54.341367985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 234 len 28672 data: 8 8 9 10 9 13 15 02/30 Read block 418 len 32768 03/30 Write block 395 len 28672 data: 6 6 8 10 16 17 20 04/30 Write block 197 len 36864 data: 11 11 9 9 8 8 9 9 7 05/30 Read block 228 len 28672 06/30 Write block 381 len 36864 data: 24 20 19 18 16 14 13 12 8 07/30 Write block 447 len 28672 data: 12 13 11 11 13 14 14 08/30 Flush 09/30 Flush 10/30 Flush 11/30 Write block 210 len 4096 data: 7 12/30 Read block 21 len 12288 13/30 Read block 26 len 32768 14/30 Read block 357 len 24576 15/30 Write block 177 len 8192 data: 11 10 16/30 Write block 359 len 24576 data: 9 9 10 10 10 13 17/30 Write block 11 len 4096 data: 15 18/30 Write block 159 len 16384 data: 9 10 11 15 19/30 Read block 168 len 28672 20/30 Read block 434 len 36864 21/30 Read block 320 len 4096 22/30 Write block 366 len 8192 data: 13 12 23/30 Read block 373 len 8192 24/30 Write block 45 len 40960 data: 10 10 7 8 6 6 5 4 7 9 25/30 Write block 235 len 40960 data: 9 10 11 10 14 16 16 18 19 16 26/30 Write block 211 len 20480 data: 6 8 10 10 6 27/30 Write block 497 len 32768 data: 12 13 11 11 13 12 11 12 28/30 Read block 239 len 40960 29/30 Write block 508 len 12288 data: 15 14 15 30/30 Write block 563 len 40960 data: 10 10 12 12 13 13 13 14 12 10 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1009)) Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:54.372751345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1768} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 237 237 237 F T T <--- 1 020-039 132 132 132 236 236 236 F F F 2 040-059 133 133 133 238 238 238 F T T <--- 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 137 137 137 245 245 245 F T T <--- 8 160-179 141 141 141 254 254 254 F T T <--- 9 180-199 143 144 144 258 259 259 F F F <--- 10 200-219 140 144 144 253 259 259 F T T <--- 11 220-239 140 144 144 253 259 259 F T T <--- 12 240-259 142 144 144 257 259 259 F T T <--- 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 143 143 143 258 258 258 F F F 16 320-339 143 143 143 258 258 258 F F F 17 340-359 142 142 142 257 257 257 F T T <--- 18 360-379 142 142 142 257 257 257 F T T <--- 19 380-399 142 144 144 256 259 259 F F F <--- 20 400-419 142 144 144 257 259 259 F F F <--- 21 420-439 142 142 142 257 257 257 F F F 22 440-459 137 144 144 245 259 259 F F F <--- 23 460-479 139 139 139 251 251 251 F F F 24 480-499 131 131 131 235 235 235 F T T <--- 25 500-519 140 140 140 253 253 253 F T T <--- 26 520-539 140 140 140 253 253 253 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 142 142 142 256 256 256 F T T <--- 29 580-599 138 138 138 250 250 250 F F F Max gen: 144, Max flush: 259 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 145 145 145 260 260 260 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 145 145 145 260 260 260 F F F 8 160-179 145 145 145 260 260 260 F F F 9 180-199 144 144 144 259 259 259 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 145 145 145 260 260 260 F F F 12 240-259 145 145 145 260 260 260 F F F 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 143 143 143 258 258 258 F F F 16 320-339 143 143 143 258 258 258 F F F 17 340-359 145 145 145 260 260 260 F F F 18 360-379 145 145 145 260 260 260 F F F 19 380-399 144 144 144 259 259 259 F F F 20 400-419 144 144 144 259 259 259 F F F 21 420-439 142 142 142 257 257 257 F F F 22 440-459 144 144 144 259 259 259 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 145 145 145 260 260 260 F F F 25 500-519 145 145 145 260 260 260 F F F 26 520-539 140 140 140 253 253 253 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 145 145 145 260 260 260 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 145, Max flush: 260 Begin loop 73 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.957644883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.975087393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.975727977Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.975803416Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.975837814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.975872673Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"Crucible ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd has session id: 3c3c8787-4e45-4df9-91d9-1088b49151eb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.976964537Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Upstairs opts: Upstairs UUID: ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977027031Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Crucible stats registered with UUID: ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.97705688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:58.977081378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977485465Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977552898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977582296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977926105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.977966955Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.97806557Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978095549Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978122512Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978149345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"2","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978181429Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978208733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:42091)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978346935Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"2","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978379089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:45824)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978407144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:35505)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.978732005Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"io task","client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.979942875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"2","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.980071539Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.980677174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.980779395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.981096491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[0]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 260, 259, 260, 260]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.981899766Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[0]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 145, 144, 145, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.981929665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.981954224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[1]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 260, 259, 260, 260]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.981978131Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[1]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 145, 144, 145, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982007739Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982033039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[2]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 260, 259, 260, 260]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982056816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[2]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 145, 144, 145, 145]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982080473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982104661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.98212481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982145722Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Next flush: 261","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982165461Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.98218521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"":"downstairs","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982206291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982225489Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"0","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.98224627Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"1","":"downstairs","session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982448179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"client":"2","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982538146Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.98261109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd is now active with session: 3c3c8787-4e45-4df9-91d9-1088b49151eb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982697791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:05:58.982766958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776,"session_id":"3c3c8787-4e45-4df9-91d9-1088b49151eb"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982837417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} {"msg":"Activated sub_volume ce24b29d-1a1a-493f-bb6b-5741ecbd4ccd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:05:58.982856735Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 362 len 40960 data: 11 11 14 13 14 13 10 9 7 7 03/30 Write block 522 len 36864 data: 13 12 13 17 12 11 12 11 11 04/30 Read block 207 len 12288 05/30 Read block 178 len 4096 06/30 Read block 236 len 40960 07/30 Write block 307 len 4096 data: 9 08/30 Read block 138 len 36864 09/30 Read block 112 len 28672 10/30 Write block 340 len 16384 data: 9 12 12 10 11/30 Read block 137 len 8192 12/30 Write block 334 len 40960 data: 11 10 12 11 11 11 10 13 13 11 13/30 Read block 253 len 32768 14/30 Write block 246 len 12288 data: 15 14 14 15/30 Write block 568 len 4096 data: 14 16/30 Flush 17/30 Write block 222 len 8192 data: 12 11 18/30 Write block 317 len 36864 data: 11 12 11 13 12 11 10 9 10 19/30 Write block 427 len 4096 data: 9 20/30 Write block 443 len 4096 data: 14 21/30 Read block 20 len 16384 22/30 Write block 482 len 16384 data: 10 12 18 14 23/30 Flush 24/30 Write block 175 len 40960 data: 9 9 12 11 10 12 12 12 15 12 25/30 Flush 26/30 Write block 453 len 20480 data: 15 13 12 11 13 27/30 Read block 394 len 36864 28/30 Read block 521 len 12288 29/30 Write block 364 len 4096 data: 15 30/30 Read block 57 len 8192 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Done Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1000)) Some(JobId(1024)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:05:59.009933697Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1776} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 145 145 145 260 260 260 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 145 145 145 260 260 260 F F F 8 160-179 146 145 146 264 260 264 F F F <--- 9 180-199 146 144 146 264 259 264 F F F <--- 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 145 146 263 260 263 F F F <--- 12 240-259 146 145 146 262 260 262 F F F <--- 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 143 146 263 258 263 F F F <--- 16 320-339 146 143 146 263 258 263 F F F <--- 17 340-359 146 145 146 262 260 262 F F F <--- 18 360-379 146 145 146 262 260 262 T T T <--- 19 380-399 144 144 144 259 259 259 F F F 20 400-419 144 144 144 259 259 259 F F F 21 420-439 146 142 146 263 257 263 F F F <--- 22 440-459 146 144 146 263 259 263 T F T <--- 23 460-479 139 139 139 251 251 251 F F F 24 480-499 146 145 146 263 260 263 F F F <--- 25 500-519 145 145 145 260 260 260 F F F 26 520-539 146 140 146 262 253 262 F F F <--- 27 540-559 140 140 140 253 253 253 F F F 28 560-579 146 145 146 262 260 262 F F F <--- 29 580-599 138 138 138 250 250 250 F F F Max gen: 146, Max flush: 264 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 132 132 132 236 236 236 F F F 2 040-059 145 145 145 260 260 260 F F F 3 060-079 141 141 141 254 254 254 F F F 4 080-099 141 141 141 254 254 254 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F F F 7 140-159 145 145 145 260 260 260 F F F 8 160-179 146 146 146 264 264 264 F F F 9 180-199 146 146 146 264 264 264 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 F F F 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 146 146 146 263 263 263 F F F 17 340-359 146 146 146 262 262 262 F F F 18 360-379 147 147 147 265 265 265 F F F 19 380-399 144 144 144 259 259 259 F F F 20 400-419 144 144 144 259 259 259 F F F 21 420-439 146 146 146 263 263 263 F F F 22 440-459 147 147 147 265 265 265 F F F 23 460-479 139 139 139 251 251 251 F F F 24 480-499 146 146 146 263 263 263 F F F 25 500-519 145 145 145 260 260 260 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 140 140 140 253 253 253 F F F 28 560-579 146 146 146 262 262 262 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 147, Max flush: 265 Begin loop 74 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.535402068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:03.552609485Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.552703571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.570334853Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.571022701Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.571119582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.571143058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.571177256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"Crucible d6febb93-7f58-484a-9eb9-a0eb945230fd has session id: 2b19f390-f311-485e-9b38-469553495367","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572049808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Upstairs opts: Upstairs UUID: d6febb93-7f58-484a-9eb9-a0eb945230fd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572108133Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Crucible stats registered with UUID: d6febb93-7f58-484a-9eb9-a0eb945230fd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572136449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:03.572167109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57269978Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572793195Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572822563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572928892Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.572959072Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"d6febb93-7f58-484a-9eb9-a0eb945230fd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573100371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573129919Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573155971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573182282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573261849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573291268Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57331778Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"ds_connection connected from Ok(127.0.0.1:64453)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573345745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"ds_connection connected from Ok(127.0.0.1:49390)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57338877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"ds_connection connected from Ok(127.0.0.1:33419)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.573847034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"io task","client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.574930502Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.575186447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57554238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.575648179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.575961747Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[0]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 264, 264, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576747137Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[0]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 146, 146, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576783499Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57681468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[1]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 264, 264, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576844028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[1]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 146, 146, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576872855Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576903596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[2]R flush_numbers[0..12]: [260, 236, 260, 254, 254, 256, 237, 260, 264, 264, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.576932693Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[2]R generation[0..12]: [145, 132, 145, 141, 141, 142, 132, 145, 146, 146, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57696155Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57699183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577056157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57708282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Next flush: 266","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577107318Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577131997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"":"downstairs","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577157848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577181224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"0","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577206824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"1","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577232745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"client":"2","":"downstairs","session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577258957Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577283596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"Wait for a query_work_queue command to finish before sending IO 2b19f390-f311-485e-9b38-469553495367"} {"msg":"d6febb93-7f58-484a-9eb9-a0eb945230fd is now active with session: 2b19f390-f311-485e-9b38-469553495367","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577328034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.57735154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784,"session_id":"2b19f390-f311-485e-9b38-469553495367"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.577376098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} {"msg":"Activated sub_volume d6febb93-7f58-484a-9eb9-a0eb945230fd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:03.5773974Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 391 len 32768 data: 6 6 7 7 7 7 9 11 02/30 Read block 240 len 20480 03/30 Flush 04/30 Write block 486 len 40960 data: 10 9 9 10 9 7 7 8 11 10 05/30 Read block 274 len 24576 06/30 Write block 555 len 24576 data: 10 10 11 11 11 8 07/30 Write block 467 len 32768 data: 10 12 10 12 14 13 14 13 08/30 Read block 273 len 16384 09/30 Flush 10/30 Write block 321 len 40960 data: 13 12 11 10 11 9 13 14 15 16 11/30 Read block 393 len 16384 12/30 Flush 13/30 Write block 78 len 28672 data: 15 14 16 19 21 19 16 14/30 Read block 266 len 20480 15/30 Write block 416 len 36864 data: 10 9 8 9 12 13 13 12 13 16/30 Read block 575 len 36864 17/30 Flush 18/30 Write block 181 len 12288 data: 13 13 16 19/30 Write block 129 len 40960 data: 7 12 9 8 8 6 6 4 8 6 20/30 Write block 484 len 24576 data: 19 15 11 10 10 11 21/30 Read block 188 len 24576 22/30 Read block 516 len 36864 23/30 Write block 26 len 28672 data: 7 11 13 16 15 11 13 24/30 Read block 171 len 4096 25/30 Write block 432 len 40960 data: 8 7 11 13 14 15 15 11 11 15 26/30 Read block 340 len 16384 27/30 Write block 492 len 36864 data: 8 9 12 11 11 13 14 12 12 28/30 Read block 388 len 8192 29/30 Read block 396 len 4096 30/30 Read block 46 len 8192 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:03.61262619Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1784} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 132 132 132 236 236 236 F T T <--- 2 040-059 145 145 145 260 260 260 F F F 3 060-079 141 148 148 254 269 269 F F F <--- 4 080-099 141 148 148 254 269 269 F F F <--- 5 100-119 142 142 142 256 256 256 F F F 6 120-139 132 132 132 237 237 237 F T T <--- 7 140-159 145 145 145 260 260 260 F F F 8 160-179 146 146 146 264 264 264 F F F 9 180-199 146 146 146 264 264 264 F T T <--- 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 F F F 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 146 148 148 263 268 268 F F F <--- 17 340-359 146 146 146 262 262 262 F F F 18 360-379 147 147 147 265 265 265 F F F 19 380-399 144 148 148 259 266 266 F F F <--- 20 400-419 144 148 148 259 269 269 F F F <--- 21 420-439 146 148 148 263 269 269 F T T <--- 22 440-459 147 147 147 265 265 265 F T T <--- 23 460-479 139 148 148 251 267 267 F F F <--- 24 480-499 146 148 148 263 267 267 F T T <--- 25 500-519 145 145 145 260 260 260 F T T <--- 26 520-539 146 146 146 262 262 262 F F F 27 540-559 140 148 148 253 267 267 F F F <--- 28 560-579 146 148 148 262 267 267 F F F <--- 29 580-599 138 138 138 250 250 250 F F F Max gen: 148, Max flush: 269 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 149 149 149 270 270 270 F F F 2 040-059 145 145 145 260 260 260 F F F 3 060-079 148 148 148 269 269 269 F F F 4 080-099 148 148 148 269 269 269 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 149 149 149 270 270 270 F F F 7 140-159 145 145 145 260 260 260 F F F 8 160-179 146 146 146 264 264 264 F F F 9 180-199 149 149 149 270 270 270 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 F F F 13 260-279 142 142 142 256 256 256 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 148 148 148 268 268 268 F F F 17 340-359 146 146 146 262 262 262 F F F 18 360-379 147 147 147 265 265 265 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 149 149 149 270 270 270 F F F 23 460-479 148 148 148 267 267 267 F F F 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 148 148 148 267 267 267 F F F 28 560-579 148 148 148 267 267 267 F F F 29 580-599 138 138 138 250 250 250 F F F Max gen: 149, Max flush: 270 Begin loop 75 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.154567777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:08.172551866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.17267066Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.190202938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.19074145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.190820917Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.19084759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.190884873Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"Crucible fc5e6f19-aab6-4064-84ba-8daebbb51f8d has session id: 2e86b7fc-424b-49f1-966f-017a303efd64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.19204373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Upstairs opts: Upstairs UUID: fc5e6f19-aab6-4064-84ba-8daebbb51f8d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.192105943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Crucible stats registered with UUID: fc5e6f19-aab6-4064-84ba-8daebbb51f8d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.192137195Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:08.192163527Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.192685717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.192743431Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.192773029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193133761Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193171706Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"fc5e6f19-aab6-4064-84ba-8daebbb51f8d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193289088Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193320179Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193348966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193377502Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193405147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193500014Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.1935332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"ds_connection connected from Ok(127.0.0.1:40053)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193653627Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"ds_connection connected from Ok(127.0.0.1:55253)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.193687975Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"ds_connection connected from Ok(127.0.0.1:59258)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.194068646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"io task","client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.195699224Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.19574794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.195936632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.196774897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.196882208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[0]R flush_numbers[0..12]: [260, 270, 260, 269, 269, 256, 270, 260, 264, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.196971354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[0]R generation[0..12]: [145, 149, 145, 148, 148, 142, 149, 145, 146, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197016333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197050651Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[1]R flush_numbers[0..12]: [260, 270, 260, 269, 269, 256, 270, 260, 264, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197082524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[1]R generation[0..12]: [145, 149, 145, 148, 148, 142, 149, 145, 146, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197113996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197147512Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[2]R flush_numbers[0..12]: [260, 270, 260, 269, 269, 256, 270, 260, 264, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197179234Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[2]R generation[0..12]: [145, 149, 145, 148, 148, 142, 149, 145, 146, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197210656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197243762Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197270555Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197299852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Next flush: 271","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197326345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197352977Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"":"downstairs","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197381003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197406613Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"0","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197438977Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"1","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197466982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"client":"2","":"downstairs","session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197495779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197522612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"fc5e6f19-aab6-4064-84ba-8daebbb51f8d is now active with session: 2e86b7fc-424b-49f1-966f-017a303efd64","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197552561Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197577751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792,"session_id":"2e86b7fc-424b-49f1-966f-017a303efd64"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197604484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} {"msg":"Activated sub_volume fc5e6f19-aab6-4064-84ba-8daebbb51f8d","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:08.197627669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 584 len 36864 data: 10 9 8 10 9 9 8 6 6 02/30 Write block 567 len 4096 data: 14 03/30 Flush 04/30 Flush 05/30 Write block 122 len 40960 data: 12 13 15 12 11 13 10 8 13 10 06/30 Flush 07/30 Write block 274 len 12288 data: 11 16 18 08/30 Read block 130 len 16384 09/30 Write block 62 len 40960 data: 9 10 9 10 12 16 18 15 13 11 10/30 Flush 11/30 Write block 148 len 32768 data: 11 8 11 12 11 12 9 9 12/30 Flush 13/30 Write block 335 len 8192 data: 11 13 14/30 Read block 506 len 32768 15/30 Read block 14 len 32768 16/30 Read block 9 len 4096 17/30 Read block 394 len 28672 18/30 Read block 357 len 20480 19/30 Read block 89 len 8192 20/30 Read block 121 len 40960 21/30 Read block 118 len 36864 22/30 Write block 578 len 8192 data: 8 11 23/30 Read block 547 len 20480 24/30 Write block 46 len 20480 data: 11 8 9 7 7 25/30 Read block 164 len 12288 26/30 Read block 30 len 28672 27/30 Write block 552 len 32768 data: 14 11 9 11 11 12 12 12 28/30 Read block 101 len 36864 29/30 Write block 170 len 16384 data: 14 15 12 9 30/30 Write block 460 len 16384 data: 8 8 8 11 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 2 2 31 Done 0 25 25 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:08.231927736Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1792} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 149 149 149 270 270 270 F F F 2 040-059 145 145 145 260 260 260 F T T <--- 3 060-079 148 150 150 269 274 274 F F F <--- 4 080-099 148 148 148 269 269 269 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 149 150 150 270 273 273 F F F <--- 7 140-159 145 150 150 260 275 275 F F F <--- 8 160-179 146 146 146 264 264 264 F T T <--- 9 180-199 149 149 149 270 270 270 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 F F F 13 260-279 142 150 150 256 274 274 F F F <--- 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 148 148 148 268 268 268 F T T <--- 17 340-359 146 146 146 262 262 262 F F F 18 360-379 147 147 147 265 265 265 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 149 149 149 270 270 270 F F F 23 460-479 148 148 148 267 267 267 F T T <--- 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 148 148 148 267 267 267 F T T <--- 28 560-579 150 150 150 271 271 271 F T T <--- 29 580-599 150 150 150 271 271 271 F F F Max gen: 150, Max flush: 275 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 F F F 1 020-039 149 149 149 270 270 270 F F F 2 040-059 151 151 151 276 276 276 F F F 3 060-079 150 150 150 274 274 274 F F F 4 080-099 148 148 148 269 269 269 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 150 150 150 275 275 275 F F F 8 160-179 151 151 151 276 276 276 F F F 9 180-199 149 149 149 270 270 270 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 F F F 13 260-279 150 150 150 274 274 274 F F F 14 280-299 142 142 142 257 257 257 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 151 151 151 276 276 276 F F F 17 340-359 146 146 146 262 262 262 F F F 18 360-379 147 147 147 265 265 265 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 149 149 149 270 270 270 F F F 23 460-479 151 151 151 276 276 276 F F F 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 151 151 151 276 276 276 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 150 150 150 271 271 271 F F F Max gen: 151, Max flush: 276 Begin loop 76 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.778997443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.801334122Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.801960017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.802111696Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.802161184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.802216483Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"Crucible ba0b7125-f9e0-4350-91b9-b67e78c9d1ff has session id: 5ac0af8c-dd4e-461c-8be5-0a7a304e6a02","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.803082081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Upstairs opts: Upstairs UUID: ba0b7125-f9e0-4350-91b9-b67e78c9d1ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.803180415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Crucible stats registered with UUID: ba0b7125-f9e0-4350-91b9-b67e78c9d1ff","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.803249161Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:12.803301153Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804085231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804249074Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804431093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804654534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804778058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"ba0b7125-f9e0-4350-91b9-b67e78c9d1ff active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804823918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804850481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804879398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.804907273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.805132878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.805361329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.80540257Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"ds_connection connected from Ok(127.0.0.1:60682)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.80555467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"ds_connection connected from Ok(127.0.0.1:57950)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.805645941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"ds_connection connected from Ok(127.0.0.1:59410)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.805938648Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"io task","client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.808073381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.808301461Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.80855973Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.809365741Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.80940557Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[0]R flush_numbers[0..12]: [260, 270, 276, 274, 269, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.809895196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[0]R generation[0..12]: [145, 149, 151, 150, 148, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810006846Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810041144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[1]R flush_numbers[0..12]: [260, 270, 276, 274, 269, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810186861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[1]R generation[0..12]: [145, 149, 151, 150, 148, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810218574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810244716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[2]R flush_numbers[0..12]: [260, 270, 276, 274, 269, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810269565Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[2]R generation[0..12]: [145, 149, 151, 150, 148, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810294394Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810332469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810354492Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810376676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Next flush: 277","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810398239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810419781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"":"downstairs","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810442035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810462105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"0","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.81048507Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1800,"client":"1","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810519017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"client":"2","":"downstairs","session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810541752Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810562583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"ba0b7125-f9e0-4350-91b9-b67e78c9d1ff is now active with session: 5ac0af8c-dd4e-461c-8be5-0a7a304e6a02","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810586079Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810606379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800,"session_id":"5ac0af8c-dd4e-461c-8be5-0a7a304e6a02"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810627741Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} {"msg":"Activated sub_volume ba0b7125-f9e0-4350-91b9-b67e78c9d1ff","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:12.810646228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 78 len 4096 data: 16 03/30 Read block 174 len 4096 04/30 Read block 208 len 28672 05/30 Read block 58 len 12288 06/30 Read block 469 len 28672 07/30 Write block 252 len 40960 data: 7 9 9 9 11 12 13 16 14 16 08/30 Read block 316 len 36864 09/30 Write block 15 len 32768 data: 14 14 13 12 10 10 11 7 10/30 Write block 35 len 4096 data: 14 11/30 Read block 326 len 28672 12/30 Read block 182 len 12288 13/30 Write block 356 len 32768 data: 13 9 7 10 10 11 12 12 14/30 Read block 356 len 4096 15/30 Write block 469 len 24576 data: 11 13 15 14 15 14 16/30 Read block 142 len 20480 17/30 Write block 453 len 36864 data: 16 14 13 12 14 14 9 9 9 18/30 Read block 531 len 28672 19/30 Read block 512 len 20480 20/30 Write block 79 len 16384 data: 15 17 20 22 21/30 Read block 422 len 32768 22/30 Write block 351 len 12288 data: 11 11 12 23/30 Read block 113 len 20480 24/30 Read block 450 len 8192 25/30 Write block 285 len 16384 data: 16 15 11 12 26/30 Write block 269 len 40960 data: 7 6 6 11 14 12 17 19 19 20 27/30 Read block 171 len 40960 28/30 Write block 363 len 24576 data: 13 16 14 15 14 11 29/30 Read block 355 len 40960 30/30 Write block 327 len 20480 data: 14 15 16 17 12 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 1 26 Done 28 5 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:12.843477887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1800} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 145 145 145 260 260 260 T F T <--- 1 020-039 149 149 149 270 270 270 T F T <--- 2 040-059 151 151 151 276 276 276 F F F 3 060-079 150 150 150 274 274 274 T T T 4 080-099 148 148 148 269 269 269 T F T <--- 5 100-119 142 142 142 256 256 256 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 150 150 150 275 275 275 F F F 8 160-179 151 151 151 276 276 276 F F F 9 180-199 149 149 149 270 270 270 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 146 146 146 262 262 262 T F T <--- 13 260-279 150 150 150 274 274 274 T F T <--- 14 280-299 142 142 142 257 257 257 T F T <--- 15 300-319 146 146 146 263 263 263 F F F 16 320-339 151 151 151 276 276 276 T F T <--- 17 340-359 146 146 146 262 262 262 T F T <--- 18 360-379 147 147 147 265 265 265 T F T <--- 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 149 149 149 270 270 270 T F T <--- 23 460-479 151 151 151 276 276 276 T F T <--- 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 151 151 151 276 276 276 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 150 150 150 271 271 271 F F F Max gen: 151, Max flush: 276 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 277 277 277 F F F 1 020-039 152 152 152 277 277 277 F F F 2 040-059 151 151 151 276 276 276 F F F 3 060-079 152 152 152 277 277 277 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 142 142 142 256 256 256 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 150 150 150 275 275 275 F F F 8 160-179 151 151 151 276 276 276 F F F 9 180-199 149 149 149 270 270 270 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 F F F 12 240-259 152 152 152 277 277 277 F F F 13 260-279 152 152 152 277 277 277 F F F 14 280-299 152 152 152 277 277 277 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 152 152 152 277 277 277 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 152 152 152 277 277 277 F F F 23 460-479 152 152 152 277 277 277 F F F 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 146 146 146 262 262 262 F F F 27 540-559 151 151 151 276 276 276 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 150 150 150 271 271 271 F F F Max gen: 152, Max flush: 277 Begin loop 77 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.390318742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.411240598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.412059063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.41216379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.412207476Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.412245812Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"Crucible 6019a911-8d55-4e7f-afc3-6c6881c15316 has session id: a9dc1b80-6c22-46bc-811b-fa8c3d91da5f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.413347295Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Upstairs opts: Upstairs UUID: 6019a911-8d55-4e7f-afc3-6c6881c15316, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.413412043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Crucible stats registered with UUID: 6019a911-8d55-4e7f-afc3-6c6881c15316","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.413443425Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:17.413470368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.413839116Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414192563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414283833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414737819Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414781255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"6019a911-8d55-4e7f-afc3-6c6881c15316 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414883186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414918466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414947463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.414975869Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415003684Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.41509299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415126667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:17.415164501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:17.415200943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:17.415238207Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:41982)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415694016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415779836Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415821528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62521)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.415856036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.417350756Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.417498307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.418252615Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:17.418359526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.417162267Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54872)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.417305309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"io task","client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.41998659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[0]R flush_numbers[0..12]: [277, 277, 276, 277, 277, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.420994098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[0]R generation[0..12]: [152, 152, 151, 152, 152, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421057544Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421089517Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[1]R flush_numbers[0..12]: [277, 277, 276, 277, 277, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421119436Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[1]R generation[0..12]: [152, 152, 151, 152, 152, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421149245Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421179785Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[2]R flush_numbers[0..12]: [277, 277, 276, 277, 277, 256, 273, 275, 276, 270, 260, 263]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421209283Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[2]R generation[0..12]: [152, 152, 151, 152, 152, 142, 150, 150, 151, 149, 145, 146]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421238701Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421269582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Max found gen is 153","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421306395Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Generation requested: 154 >= found:153","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421333538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Next flush: 278","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421358678Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421383337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"":"downstairs","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421409408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421433345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"0","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421459467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"1","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421485368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"client":"2","":"downstairs","session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421512Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"Wait for a query_work_queue command to finish before sending IO session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421546749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"6019a911-8d55-4e7f-afc3-6c6881c15316 is now active with session: a9dc1b80-6c22-46bc-811b-fa8c3d91da5f","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421574945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421598371Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808,"session_id":"a9dc1b80-6c22-46bc-811b-fa8c3d91da5f"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.42162323Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} {"msg":"Activated sub_volume 6019a911-8d55-4e7f-afc3-6c6881c15316","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:27.421644953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 527 len 40960 data: 12 13 12 12 11 11 12 9 9 11 02/30 Read block 286 len 4096 03/30 Read block 299 len 12288 04/30 Read block 517 len 40960 05/30 Flush 06/30 Write block 320 len 8192 data: 14 14 07/30 Read block 20 len 16384 08/30 Write block 107 len 32768 data: 7 8 10 13 12 15 15 13 09/30 Write block 265 len 28672 data: 10 10 10 9 8 7 7 10/30 Write block 191 len 4096 data: 11 11/30 Write block 7 len 20480 data: 7 11 13 16 16 12/30 Read block 504 len 28672 13/30 Write block 235 len 16384 data: 10 11 12 11 14/30 Read block 154 len 12288 15/30 Read block 152 len 40960 16/30 Read block 56 len 40960 17/30 Read block 374 len 28672 18/30 Write block 20 len 24576 data: 11 12 8 7 7 7 19/30 Read block 346 len 12288 20/30 Read block 254 len 24576 21/30 Write block 546 len 24576 data: 10 11 10 14 13 13 22/30 Read block 489 len 28672 23/30 Write block 178 len 36864 data: 12 11 13 14 14 17 13 13 13 24/30 Read block 481 len 4096 25/30 Write block 227 len 32768 data: 11 11 9 9 10 10 5 9 26/30 Write block 320 len 24576 data: 15 15 13 12 11 12 27/30 Write block 110 len 28672 data: 14 13 16 16 14 14 16 28/30 Write block 594 len 12288 data: 6 5 5 29/30 Write block 142 len 12288 data: 7 8 11 30/30 Write block 337 len 4096 data: 12 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 8 Sent Done Sent false 1025 Acked Write 6 Sent Sent Sent false 1026 Acked Write 7 Sent Sent Sent false 1027 Acked Write 3 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 6 5 28 39 Done 24 25 2 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1004)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:27.449209428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1808} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 277 277 277 T T F <--- 1 020-039 152 152 152 277 277 277 T T F <--- 2 040-059 151 151 151 276 276 276 F F F 3 060-079 152 152 152 277 277 277 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 142 142 142 256 256 256 T T F <--- 6 120-139 150 150 150 273 273 273 F F F 7 140-159 150 150 150 275 275 275 F T F <--- 8 160-179 151 151 151 276 276 276 T T F <--- 9 180-199 149 149 149 270 270 270 T T F <--- 10 200-219 145 145 145 260 260 260 F F F 11 220-239 146 146 146 263 263 263 T T F <--- 12 240-259 152 152 152 277 277 277 F F F 13 260-279 152 152 152 277 277 277 T T F <--- 14 280-299 152 152 152 277 277 277 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 152 152 152 277 277 277 T T F <--- 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 152 152 152 277 277 277 F F F 23 460-479 152 152 152 277 277 277 F F F 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 154 154 146 278 278 262 F F T <--- 27 540-559 151 151 151 276 276 276 T T F <--- 28 560-579 151 151 151 276 276 276 F F F 29 580-599 150 150 150 271 271 271 F T F <--- Max gen: 154, Max flush: 278 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 339a340 > 12, 342d342 < 11, diff found after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 279 279 279 F F F 1 020-039 155 155 155 279 279 279 F F F 2 040-059 151 151 151 276 276 276 F F F 3 060-079 152 152 152 277 277 277 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 155 155 155 279 279 279 F F F 9 180-199 155 155 155 279 279 279 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 155 155 155 279 279 279 F F F 12 240-259 152 152 152 277 277 277 F F F 13 260-279 155 155 155 279 279 279 F F F 14 280-299 152 152 152 277 277 277 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 149 149 149 270 270 270 F F F 22 440-459 152 152 152 277 277 277 F F F 23 460-479 152 152 152 277 277 277 F F F 24 480-499 149 149 149 270 270 270 F F F 25 500-519 149 149 149 270 270 270 F F F 26 520-539 154 154 154 278 278 278 F F F 27 540-559 155 155 155 279 279 279 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 155 155 155 279 279 279 F F F Max gen: 155, Max flush: 279 Begin loop 78 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.01286781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.033446679Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.034041774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.034122493Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.034147893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.034184796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"Crucible cf5f1aeb-70e8-4428-8623-6999ced773e8 has session id: 57c682d6-b983-4244-b799-bdc13b047149","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.035257473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Upstairs opts: Upstairs UUID: cf5f1aeb-70e8-4428-8623-6999ced773e8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.035336689Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Crucible stats registered with UUID: cf5f1aeb-70e8-4428-8623-6999ced773e8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.03536283Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:32.035384022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.035880542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.035933146Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.035956913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036317464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.03635578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"cf5f1aeb-70e8-4428-8623-6999ced773e8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036571686Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036602958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036628308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036650391Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036672816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036804415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.036834605Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"ds_connection connected from Ok(127.0.0.1:62158)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.037074999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"ds_connection connected from Ok(127.0.0.1:36751)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.037336114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"ds_connection connected from Ok(127.0.0.1:53113)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.037379721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.039293858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.039354317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.039403835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040139406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040223502Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[0]R flush_numbers[0..12]: [279, 279, 276, 277, 277, 279, 273, 279, 279, 279, 260, 279]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040462083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[0]R generation[0..12]: [155, 155, 151, 152, 152, 155, 150, 155, 155, 155, 145, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040510058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040542161Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[1]R flush_numbers[0..12]: [279, 279, 276, 277, 277, 279, 273, 279, 279, 279, 260, 279]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040572571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[1]R generation[0..12]: [155, 155, 151, 152, 152, 155, 150, 155, 155, 155, 145, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040602049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.04063254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[2]R flush_numbers[0..12]: [279, 279, 276, 277, 277, 279, 273, 279, 279, 279, 260, 279]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040662228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[2]R generation[0..12]: [155, 155, 151, 152, 152, 155, 150, 155, 155, 155, 145, 155]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040691867Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040722607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040747757Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040773858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040798407Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040823306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"":"downstairs","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040849438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040873214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"0","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040899707Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"1","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040929045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"client":"2","":"downstairs","session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040955898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} Wait for a query_work_queue command to finish before sending IO {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.040993953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"cf5f1aeb-70e8-4428-8623-6999ced773e8 is now active with session: 57c682d6-b983-4244-b799-bdc13b047149","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.04102307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.041047078Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819,"session_id":"57c682d6-b983-4244-b799-bdc13b047149"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.041071796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} {"msg":"Activated sub_volume cf5f1aeb-70e8-4428-8623-6999ced773e8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:32.041093239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 160 len 20480 data: 11 12 16 14 13 02/30 Read block 35 len 20480 03/30 Read block 62 len 36864 04/30 Write block 225 len 40960 data: 12 13 12 12 10 10 11 11 6 10 05/30 Read block 373 len 16384 06/30 Flush 07/30 Write block 582 len 8192 data: 8 10 08/30 Write block 547 len 12288 data: 12 11 15 09/30 Read block 586 len 40960 10/30 Read block 350 len 20480 11/30 Read block 466 len 32768 12/30 Write block 420 len 32768 data: 13 14 14 13 14 8 6 10 13/30 Read block 393 len 40960 14/30 Write block 526 len 4096 data: 13 15/30 Write block 510 len 24576 data: 16 14 8 8 8 7 16/30 Write block 0 len 20480 data: 2 4 6 7 8 17/30 Read block 543 len 32768 18/30 Write block 29 len 20480 data: 17 16 12 14 12 19/30 Read block 580 len 16384 20/30 Read block 553 len 20480 21/30 Flush 22/30 Write block 527 len 36864 data: 13 14 13 13 12 12 13 10 10 23/30 Read block 248 len 36864 24/30 Read block 569 len 8192 25/30 Read block 105 len 12288 26/30 Read block 176 len 24576 27/30 Read block 209 len 4096 28/30 Write block 61 len 20480 data: 10 10 11 10 11 29/30 Write block 484 len 20480 data: 20 16 12 11 11 30/30 Read block 139 len 24576 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 2 Done Done Done false 1007 Acked Write 3 Done Done Done false 1008 Acked Read 10 Done Done Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 5 Done Sent Sent false 1028 Acked Write 5 Done Sent Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 21 3 24 Done 24 3 21 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1005)) Some(JobId(1020)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:32.072484645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1819} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 155 156 281 279 281 F F F <--- 1 020-039 156 155 156 281 279 281 F F F <--- 2 040-059 151 151 151 276 276 276 F F F 3 060-079 152 152 152 277 277 277 T F T <--- 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 156 156 156 280 280 280 F F F 9 180-199 155 155 155 279 279 279 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 156 156 156 280 280 280 F F F 12 240-259 152 152 152 277 277 277 F F F 13 260-279 155 155 155 279 279 279 F F F 14 280-299 152 152 152 277 277 277 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 156 149 156 281 270 281 F F F <--- 22 440-459 152 152 152 277 277 277 F F F 23 460-479 152 152 152 277 277 277 F F F 24 480-499 149 149 149 270 270 270 T F T <--- 25 500-519 156 149 156 281 270 281 F F F <--- 26 520-539 156 154 156 281 278 281 T F T <--- 27 540-559 156 155 156 281 279 281 F T F <--- 28 560-579 151 151 151 276 276 276 F F F 29 580-599 156 155 156 281 279 281 F T F <--- Max gen: 156, Max flush: 281 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 281 281 281 F F F 1 020-039 156 156 156 281 281 281 F F F 2 040-059 151 151 151 276 276 276 F F F 3 060-079 157 157 157 282 282 282 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 F F F 6 120-139 150 150 150 273 273 273 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 156 156 156 280 280 280 F F F 9 180-199 155 155 155 279 279 279 F F F 10 200-219 145 145 145 260 260 260 F F F 11 220-239 156 156 156 280 280 280 F F F 12 240-259 152 152 152 277 277 277 F F F 13 260-279 155 155 155 279 279 279 F F F 14 280-299 152 152 152 277 277 277 F F F 15 300-319 146 146 146 263 263 263 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 156 156 156 281 281 281 F F F 22 440-459 152 152 152 277 277 277 F F F 23 460-479 152 152 152 277 277 277 F F F 24 480-499 157 157 157 282 282 282 F F F 25 500-519 156 156 156 281 281 281 F F F 26 520-539 157 157 157 282 282 282 F F F 27 540-559 156 156 156 281 281 281 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 156 156 156 281 281 281 F F F Max gen: 157, Max flush: 282 Begin loop 79 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.57298815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.5920878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59253839Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59263481Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.592666011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.592698405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"Crucible 1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa has session id: 24f42a5d-106a-4e78-a0d3-dc27afb47a44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.593580516Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Upstairs opts: Upstairs UUID: 1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.593869256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Crucible stats registered with UUID: 1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.593912952Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:36.59393731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.594323693Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59439356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.594422067Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.594886443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.594914078Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595164392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59519413Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595218939Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595242927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595363464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595393143Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595417912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"ds_connection connected from Ok(127.0.0.1:48833)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59578711Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"ds_connection connected from Ok(127.0.0.1:46490)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.595987556Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"ds_connection connected from Ok(127.0.0.1:56387)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.596040591Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.597629486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.597879209Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.598158801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.598331501Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59843191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[0]R flush_numbers[0..12]: [281, 281, 276, 282, 277, 279, 273, 279, 280, 279, 260, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.598906226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[0]R generation[0..12]: [156, 156, 151, 157, 152, 155, 150, 155, 156, 155, 145, 156]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.598947176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.598976925Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[1]R flush_numbers[0..12]: [281, 281, 276, 282, 277, 279, 273, 279, 280, 279, 260, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59900495Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[1]R generation[0..12]: [156, 156, 151, 157, 152, 155, 150, 155, 156, 155, 145, 156]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599032876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599061242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[2]R flush_numbers[0..12]: [281, 281, 276, 282, 277, 279, 273, 279, 280, 279, 260, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599088656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[2]R generation[0..12]: [156, 156, 151, 157, 152, 155, 150, 155, 156, 155, 145, 156]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59911598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599144195Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599175086Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599199745Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Next flush: 283","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59922282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599245846Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"":"downstairs","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599270404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599293239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"0","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599317678Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"1","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599341925Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"client":"2","":"downstairs","session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599366484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599389529Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa is now active with session: 24f42a5d-106a-4e78-a0d3-dc27afb47a44","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.59951159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599533674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827,"session_id":"24f42a5d-106a-4e78-a0d3-dc27afb47a44"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599556739Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} {"msg":"Activated sub_volume 1c2a4283-87c5-4ee0-97fe-d6e8f59e3efa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:36.599576578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 315 len 16384 data: 16 13 12 13 02/30 Write block 447 len 20480 data: 13 14 12 12 14 03/30 Read block 242 len 4096 04/30 Write block 217 len 32768 data: 10 11 14 13 12 13 12 11 05/30 Read block 285 len 24576 06/30 Write block 247 len 20480 data: 15 15 11 9 8 07/30 Write block 168 len 24576 data: 12 13 15 16 13 10 08/30 Read block 564 len 20480 09/30 Read block 476 len 24576 10/30 Write block 204 len 8192 data: 10 8 11/30 Write block 256 len 32768 data: 12 13 14 17 15 17 16 12 12/30 Flush 13/30 Write block 218 len 32768 data: 12 15 14 13 14 13 12 13 14/30 Read block 44 len 8192 15/30 Write block 551 len 24576 data: 14 15 12 10 12 12 16/30 Read block 494 len 12288 17/30 Write block 444 len 28672 data: 11 14 14 14 15 13 13 18/30 Read block 326 len 4096 19/30 Read block 150 len 12288 20/30 Flush 21/30 Write block 122 len 28672 data: 13 14 16 13 12 14 11 22/30 Read block 276 len 28672 23/30 Write block 28 len 32768 data: 14 18 17 13 15 13 14 15 24/30 Read block 50 len 8192 25/30 Flush 26/30 Write block 73 len 24576 data: 12 14 16 17 18 17 27/30 Write block 184 len 28672 data: 14 14 14 12 12 11 12 28/30 Write block 475 len 8192 data: 14 12 29/30 Write block 297 len 28672 data: 19 16 13 13 10 14 13 30/30 Write block 1 len 24576 data: 5 7 8 9 8 9 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Sent Sent Sent false 1026 Acked Write 7 Sent Sent Sent false 1027 Acked Write 2 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 5 5 24 34 Done 25 25 6 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:36.625074786Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1827} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 281 281 281 T T F <--- 1 020-039 158 158 156 285 285 281 F F F <--- 2 040-059 151 151 151 276 276 276 F F F 3 060-079 157 157 157 282 282 282 T T F <--- 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 F F F 6 120-139 158 158 150 285 285 273 F F F <--- 7 140-159 155 155 155 279 279 279 F F F 8 160-179 158 158 156 283 283 280 F F F <--- 9 180-199 155 155 155 279 279 279 T T F <--- 10 200-219 158 158 145 284 284 260 F F T <--- 11 220-239 158 158 156 284 284 280 F F T <--- 12 240-259 158 158 152 283 283 277 F F T <--- 13 260-279 158 158 155 283 283 279 F F F <--- 14 280-299 152 152 152 277 277 277 T T F <--- 15 300-319 158 158 146 283 283 263 T T T <--- 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 156 156 156 281 281 281 F F F 22 440-459 158 158 152 284 284 277 F F T <--- 23 460-479 152 152 152 277 277 277 T T F <--- 24 480-499 157 157 157 282 282 282 F F F 25 500-519 156 156 156 281 281 281 F F F 26 520-539 157 157 157 282 282 282 F F F 27 540-559 158 158 156 284 284 281 F F F <--- 28 560-579 151 151 151 276 276 276 F F F 29 580-599 156 156 156 281 281 281 F F F Max gen: 158, Max flush: 285 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 159 159 159 286 286 286 F F F 1 020-039 158 158 158 285 285 285 F F F 2 040-059 151 151 151 276 276 276 F F F 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 158 158 158 283 283 283 F F F 9 180-199 159 159 159 286 286 286 F F F 10 200-219 158 158 158 284 284 284 F F F 11 220-239 158 158 158 284 284 284 F F F 12 240-259 158 158 158 283 283 283 F F F 13 260-279 158 158 158 283 283 283 F F F 14 280-299 159 159 159 286 286 286 F F F 15 300-319 159 159 159 286 286 286 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 F F F 18 360-379 152 152 152 277 277 277 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 148 148 148 269 269 269 F F F 21 420-439 156 156 156 281 281 281 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 159 159 159 286 286 286 F F F 24 480-499 157 157 157 282 282 282 F F F 25 500-519 156 156 156 281 281 281 F F F 26 520-539 157 157 157 282 282 282 F F F 27 540-559 158 158 158 284 284 284 F F F 28 560-579 151 151 151 276 276 276 F F F 29 580-599 156 156 156 281 281 281 F F F Max gen: 159, Max flush: 286 Begin loop 80 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.19673094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.217706673Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.218197752Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.2182676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.218289213Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.218317749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"Crucible 3309163e-6d03-483e-9ee4-64eeb05152d6 has session id: f71a33c7-48e3-4de1-b386-2a89f2a8fdd9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.219439022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Upstairs opts: Upstairs UUID: 3309163e-6d03-483e-9ee4-64eeb05152d6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.219557716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Crucible stats registered with UUID: 3309163e-6d03-483e-9ee4-64eeb05152d6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.21959035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:41.219615049Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220094876Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220196697Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220227808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220369658Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220405619Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"3309163e-6d03-483e-9ee4-64eeb05152d6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220525456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220561717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220591476Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220619211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220646324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220731452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220909794Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"ds_connection connected from Ok(127.0.0.1:52288)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.220955855Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:41.220987728Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:41.22104973Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"ds_connection connected from Ok(127.0.0.1:45417)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.221073758Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:41.221532713Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.221655726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.221690695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.222710898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.222852147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.223442141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:41.22349195Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.223047506Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"ds_connection connected from Ok(127.0.0.1:51302)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.223651478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.22568949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[0]R flush_numbers[0..12]: [286, 285, 276, 286, 277, 279, 285, 279, 283, 286, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226506192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[0]R generation[0..12]: [159, 158, 151, 159, 152, 155, 158, 155, 158, 159, 158, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226552152Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226584947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[1]R flush_numbers[0..12]: [286, 285, 276, 286, 277, 279, 285, 279, 283, 286, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226615287Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[1]R generation[0..12]: [159, 158, 151, 159, 152, 155, 158, 155, 158, 159, 158, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226644595Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226675035Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[2]R flush_numbers[0..12]: [286, 285, 276, 286, 277, 279, 285, 279, 283, 286, 284, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226704423Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[2]R generation[0..12]: [159, 158, 151, 159, 152, 155, 158, 155, 158, 159, 158, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226734192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226765083Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226798328Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.22682475Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Next flush: 287","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226849459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226874148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"":"downstairs","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226899969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226923225Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"0","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226948995Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"1","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.226974786Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"client":"2","":"downstairs","session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227000778Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227025967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"3309163e-6d03-483e-9ee4-64eeb05152d6 is now active with session: f71a33c7-48e3-4de1-b386-2a89f2a8fdd9","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227053682Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227076858Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835,"session_id":"f71a33c7-48e3-4de1-b386-2a89f2a8fdd9"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227101577Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} {"msg":"Activated sub_volume 3309163e-6d03-483e-9ee4-64eeb05152d6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:51.227123099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 417 len 24576 data: 10 9 10 14 15 15 02/30 Flush 03/30 Write block 160 len 4096 data: 12 04/30 Write block 101 len 16384 data: 10 10 11 11 05/30 Write block 28 len 12288 data: 15 19 18 06/30 Write block 541 len 4096 data: 14 07/30 Read block 448 len 24576 08/30 Write block 542 len 12288 data: 16 13 12 09/30 Read block 64 len 16384 10/30 Read block 540 len 24576 11/30 Read block 506 len 28672 12/30 Write block 371 len 32768 data: 8 6 10 12 13 12 16 19 13/30 Write block 341 len 32768 data: 14 14 12 9 8 12 13 12 14/30 Read block 476 len 36864 15/30 Read block 38 len 24576 16/30 Read block 421 len 8192 17/30 Read block 1 len 20480 18/30 Write block 560 len 32768 data: 9 8 9 11 11 13 13 15 19/30 Write block 105 len 36864 data: 12 10 8 9 11 15 14 17 17 20/30 Read block 314 len 24576 21/30 Read block 477 len 28672 22/30 Read block 369 len 36864 23/30 Read block 480 len 36864 24/30 Write block 252 len 12288 data: 8 10 10 25/30 Write block 197 len 40960 data: 12 12 10 10 9 9 10 11 9 10 26/30 Write block 415 len 20480 data: 10 11 11 10 11 27/30 Write block 45 len 28672 data: 11 12 9 10 8 8 6 28/30 Read block 329 len 24576 29/30 Write block 356 len 24576 data: 14 10 8 11 11 12 30/30 Read block 356 len 40960 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 28 27 0 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:51.257086888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1835} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 159 159 159 286 286 286 F F F 1 020-039 158 158 158 285 285 285 T T F <--- 2 040-059 151 151 151 276 276 276 T T F <--- 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 155 155 155 279 279 279 T T F <--- 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 158 158 158 283 283 283 T T F <--- 9 180-199 159 159 159 286 286 286 T T F <--- 10 200-219 158 158 158 284 284 284 T T F <--- 11 220-239 158 158 158 284 284 284 F F F 12 240-259 158 158 158 283 283 283 T T F <--- 13 260-279 158 158 158 283 283 283 F F F 14 280-299 159 159 159 286 286 286 F F F 15 300-319 159 159 159 286 286 286 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 152 152 152 277 277 277 T T F <--- 18 360-379 152 152 152 277 277 277 T T F <--- 19 380-399 148 148 148 266 266 266 F F F 20 400-419 160 160 160 287 287 287 T T F <--- 21 420-439 160 160 160 287 287 287 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 159 159 159 286 286 286 F F F 24 480-499 157 157 157 282 282 282 F F F 25 500-519 156 156 156 281 281 281 F F F 26 520-539 157 157 157 282 282 282 F F F 27 540-559 158 158 158 284 284 284 T T F <--- 28 560-579 151 151 151 276 276 276 T T F <--- 29 580-599 156 156 156 281 281 281 F F F Max gen: 160, Max flush: 287 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 159 159 159 286 286 286 F F F 1 020-039 161 161 161 288 288 288 F F F 2 040-059 161 161 161 288 288 288 F F F 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 161 161 161 288 288 288 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 158 158 158 284 284 284 F F F 12 240-259 161 161 161 288 288 288 F F F 13 260-279 158 158 158 283 283 283 F F F 14 280-299 159 159 159 286 286 286 F F F 15 300-319 159 159 159 286 286 286 F F F 16 320-339 155 155 155 279 279 279 F F F 17 340-359 161 161 161 288 288 288 F F F 18 360-379 161 161 161 288 288 288 F F F 19 380-399 148 148 148 266 266 266 F F F 20 400-419 161 161 161 288 288 288 F F F 21 420-439 160 160 160 287 287 287 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 159 159 159 286 286 286 F F F 24 480-499 157 157 157 282 282 282 F F F 25 500-519 156 156 156 281 281 281 F F F 26 520-539 157 157 157 282 282 282 F F F 27 540-559 161 161 161 288 288 288 F F F 28 560-579 161 161 161 288 288 288 F F F 29 580-599 156 156 156 281 281 281 F F F Max gen: 161, Max flush: 288 Begin loop 81 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.797010082Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.821736366Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82247815Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82256944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.822598427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.822641171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"Crucible bf89dae1-c843-45e0-893d-9633f54f97f6 has session id: 83de276c-b80a-4cfe-ab2f-26a98bb5e6ee","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.823949683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Upstairs opts: Upstairs UUID: bf89dae1-c843-45e0-893d-9633f54f97f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824017988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Crucible stats registered with UUID: bf89dae1-c843-45e0-893d-9633f54f97f6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824051374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:55.824080642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824565779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824681427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824718851Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824753309Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824813458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"bf89dae1-c843-45e0-893d-9633f54f97f6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.824928285Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82496647Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82543249Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825567726Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825604068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825635781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:38254)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825692252Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825724015Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:40244)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.825957496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:33395)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.826017644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.827203915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82735822Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.827400623Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.827953615Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82806788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[0]R flush_numbers[0..12]: [286, 288, 288, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828220902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[0]R generation[0..12]: [159, 161, 161, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828262354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828306882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[1]R flush_numbers[0..12]: [286, 288, 288, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828342402Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[1]R generation[0..12]: [159, 161, 161, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.82837713Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828413192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[2]R flush_numbers[0..12]: [286, 288, 288, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828449674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[2]R generation[0..12]: [159, 161, 161, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828486316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828522497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828552186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828582866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Next flush: 289","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828611934Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828641101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"":"downstairs","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828672002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:06:55.828700438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"0","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828738213Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"1","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828768924Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"client":"2","":"downstairs","session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828806097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828836878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"bf89dae1-c843-45e0-893d-9633f54f97f6 is now active with session: 83de276c-b80a-4cfe-ab2f-26a98bb5e6ee","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828869893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828897547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843,"session_id":"83de276c-b80a-4cfe-ab2f-26a98bb5e6ee"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828927907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} {"msg":"Activated sub_volume bf89dae1-c843-45e0-893d-9633f54f97f6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:06:55.828953107Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 479 len 40960 data: 11 11 10 11 13 21 17 13 12 12 02/30 Read block 470 len 28672 03/30 Write block 363 len 20480 data: 14 17 15 16 15 04/30 Read block 571 len 8192 05/30 Flush 06/30 Write block 395 len 24576 data: 8 8 10 12 17 18 07/30 Read block 423 len 28672 08/30 Read block 580 len 16384 09/30 Flush 10/30 Write block 332 len 16384 data: 10 10 12 12 11/30 Read block 262 len 24576 12/30 Write block 462 len 20480 data: 9 12 10 13 12 13/30 Read block 208 len 4096 14/30 Read block 260 len 20480 15/30 Write block 515 len 24576 data: 8 6 7 8 13 14 16/30 Read block 300 len 12288 17/30 Read block 93 len 24576 18/30 Write block 470 len 24576 data: 14 16 15 16 15 15 19/30 Read block 299 len 32768 20/30 Read block 41 len 12288 21/30 Write block 50 len 4096 data: 9 22/30 Read block 375 len 36864 23/30 Write block 304 len 20480 data: 7 8 9 10 11 24/30 Read block 171 len 32768 25/30 Flush 26/30 Write block 409 len 24576 data: 10 12 13 14 10 12 27/30 Read block 399 len 8192 28/30 Write block 595 len 16384 data: 6 6 6 4 29/30 Write block 520 len 8192 data: 15 14 30/30 Write block 18 len 40960 data: 13 11 12 13 9 8 8 8 8 12 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 4 Done Done Done false 1010 Acked Read 6 Done Done Done false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 19 25 Done 18 18 2 38 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) Some(JobId(1008)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:06:55.859809408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1843} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 159 159 159 286 286 286 T T F <--- 1 020-039 161 161 161 288 288 288 T T F <--- 2 040-059 162 162 161 291 291 288 F F F <--- 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 161 161 161 288 288 288 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 158 158 158 284 284 284 F F F 12 240-259 161 161 161 288 288 288 F F F 13 260-279 158 158 158 283 283 283 F F F 14 280-299 159 159 159 286 286 286 F F F 15 300-319 162 162 159 291 291 286 F F F <--- 16 320-339 162 162 155 291 291 279 F F T <--- 17 340-359 161 161 161 288 288 288 F F F 18 360-379 162 162 162 289 289 289 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 162 162 162 290 290 290 T T F <--- 21 420-439 160 160 160 287 287 287 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 162 162 162 291 291 289 F F F <--- 24 480-499 162 162 162 289 289 289 F F F 25 500-519 162 162 156 291 291 281 F F F <--- 26 520-539 162 162 157 291 291 282 T T F <--- 27 540-559 161 161 161 288 288 288 F F F 28 560-579 161 161 161 288 288 288 F F F 29 580-599 156 156 156 281 281 281 T T F <--- Max gen: 162, Max flush: 291 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 163 163 163 292 292 292 F F F 1 020-039 163 163 163 292 292 292 F F F 2 040-059 162 162 162 291 291 291 F F F 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 161 161 161 288 288 288 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 158 158 158 284 284 284 F F F 12 240-259 161 161 161 288 288 288 F F F 13 260-279 158 158 158 283 283 283 F F F 14 280-299 159 159 159 286 286 286 F F F 15 300-319 162 162 162 291 291 291 F F F 16 320-339 162 162 162 291 291 291 F F F 17 340-359 161 161 161 288 288 288 F F F 18 360-379 162 162 162 289 289 289 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 163 163 163 292 292 292 F F F 21 420-439 160 160 160 287 287 287 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 162 162 162 291 291 291 F F F 24 480-499 162 162 162 289 289 289 F F F 25 500-519 162 162 162 291 291 291 F F F 26 520-539 163 163 163 292 292 292 F F F 27 540-559 161 161 161 288 288 288 F F F 28 560-579 161 161 161 288 288 288 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 163, Max flush: 292 Begin loop 82 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.39813491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.421467904Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.422341398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.422427809Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.422455223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.422502386Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"Crucible d6276a86-d032-4fc4-bbce-0686ed1f1e72 has session id: 8367dd5a-8bc2-4d9a-9797-e04cd9f36e15","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.423354447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Upstairs opts: Upstairs UUID: d6276a86-d032-4fc4-bbce-0686ed1f1e72, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.423424335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Crucible stats registered with UUID: d6276a86-d032-4fc4-bbce-0686ed1f1e72","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.423458412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:00.42348768Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.423755829Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.423932358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.424067244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.424821262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.42486596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"d6276a86-d032-4fc4-bbce-0686ed1f1e72 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425015094Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425047638Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425079181Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425110262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425140732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425172425Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"ds_connection connected from Ok(127.0.0.1:40787)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425392619Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425458569Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"ds_connection connected from Ok(127.0.0.1:33613)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425687912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"ds_connection connected from Ok(127.0.0.1:52027)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.425734814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.426851999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.426947027Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.427463816Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.427833586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.428077437Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[0]R flush_numbers[0..12]: [292, 292, 291, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.42919341Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[0]R generation[0..12]: [163, 163, 162, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429286644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429332254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[1]R flush_numbers[0..12]: [292, 292, 291, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429369808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[1]R generation[0..12]: [163, 163, 162, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429403003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429438002Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[2]R flush_numbers[0..12]: [292, 292, 291, 286, 277, 288, 285, 279, 288, 288, 288, 284]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429474334Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"[2]R generation[0..12]: [163, 163, 162, 159, 152, 161, 158, 155, 161, 161, 161, 158]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429506016Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429549352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429582618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429631524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Next flush: 293","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429660471Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429696342Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"":"downstairs","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429724858Z","hostname":"bmat-EVT22200007-0000ec3a","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429761611Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"0","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.42979681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"1","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429825888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"client":"2","":"downstairs","session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.42985737Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429891818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"d6276a86-d032-4fc4-bbce-0686ed1f1e72 is now active with session: 8367dd5a-8bc2-4d9a-9797-e04cd9f36e15","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429929011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429960123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851,"session_id":"8367dd5a-8bc2-4d9a-9797-e04cd9f36e15"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.429987216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} {"msg":"Activated sub_volume d6276a86-d032-4fc4-bbce-0686ed1f1e72","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:00.430010923Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 236 len 28672 data: 12 13 12 15 17 17 19 02/30 Read block 478 len 16384 03/30 Write block 521 len 32768 data: 15 14 13 14 18 14 14 15 04/30 Read block 447 len 12288 05/30 Read block 196 len 24576 06/30 Write block 411 len 28672 data: 14 15 11 13 11 12 12 07/30 Read block 345 len 4096 08/30 Write block 275 len 16384 data: 18 20 20 21 09/30 Read block 428 len 24576 10/30 Write block 223 len 4096 data: 14 11/30 Write block 572 len 20480 data: 11 10 10 9 11 12/30 Read block 265 len 4096 13/30 Read block 456 len 20480 14/30 Write block 308 len 24576 data: 12 11 10 13 13 14 15/30 Flush 16/30 Write block 37 len 40960 data: 14 14 10 8 6 6 8 8 12 13 17/30 Write block 104 len 8192 data: 12 13 18/30 Write block 429 len 12288 data: 10 11 11 19/30 Write block 273 len 32768 data: 15 13 19 21 21 22 18 19 20/30 Read block 187 len 36864 21/30 Write block 460 len 40960 data: 10 10 10 13 11 14 13 11 13 12 22/30 Read block 77 len 40960 23/30 Read block 392 len 28672 24/30 Write block 542 len 12288 data: 17 14 13 25/30 Read block 255 len 16384 26/30 Write block 460 len 32768 data: 11 11 11 14 12 15 14 12 27/30 Read block 585 len 8192 28/30 Read block 284 len 12288 29/30 Read block 275 len 4096 30/30 Read block 103 len 40960 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 1 27 Done 30 4 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:00.463286718Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1851} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 163 163 163 292 292 292 F F F 1 020-039 163 163 163 292 292 292 T F T <--- 2 040-059 162 162 162 291 291 291 T F T <--- 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 161 161 161 288 288 288 T F T <--- 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 158 164 293 284 293 F T F <--- 12 240-259 164 161 164 293 288 293 F T F <--- 13 260-279 164 158 164 293 283 293 T F T <--- 14 280-299 159 159 159 286 286 286 T F T <--- 15 300-319 164 162 164 293 291 293 F F F <--- 16 320-339 162 162 162 291 291 291 F F F 17 340-359 161 161 161 288 288 288 F F F 18 360-379 162 162 162 289 289 289 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 164 163 164 293 292 293 F F F <--- 21 420-439 160 160 160 287 287 287 T F T <--- 22 440-459 158 158 158 284 284 284 F F F 23 460-479 162 162 162 291 291 291 T F T <--- 24 480-499 162 162 162 289 289 289 F F F 25 500-519 162 162 162 291 291 291 F F F 26 520-539 164 163 164 293 292 293 F T F <--- 27 540-559 161 161 161 288 288 288 T F T <--- 28 560-579 164 161 164 293 288 293 F F F <--- 29 580-599 163 163 163 292 292 292 F F F Max gen: 164, Max flush: 293 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 163 163 163 292 292 292 F F F 1 020-039 165 165 165 294 294 294 F F F 2 040-059 165 165 165 294 294 294 F F F 3 060-079 159 159 159 286 286 286 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 155 155 155 279 279 279 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 165 165 165 294 294 294 F F F 14 280-299 165 165 165 294 294 294 F F F 15 300-319 164 164 164 293 293 293 F F F 16 320-339 162 162 162 291 291 291 F F F 17 340-359 161 161 161 288 288 288 F F F 18 360-379 162 162 162 289 289 289 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 164 164 164 293 293 293 F F F 21 420-439 165 165 165 294 294 294 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 165 165 165 294 294 294 F F F 24 480-499 162 162 162 289 289 289 F F F 25 500-519 162 162 162 291 291 291 F F F 26 520-539 164 164 164 293 293 293 F F F 27 540-559 165 165 165 294 294 294 F F F 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 165, Max flush: 294 Begin loop 83 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.041660972Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06214789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.062651293Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.062721913Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.062751982Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.062787582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"Crucible 31b8587b-02b9-46a0-a348-3e7e8ba84169 has session id: 7cb832d2-cf33-4eb1-ad75-ab96cd681da3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.064102667Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Upstairs opts: Upstairs UUID: 31b8587b-02b9-46a0-a348-3e7e8ba84169, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06415997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Crucible stats registered with UUID: 31b8587b-02b9-46a0-a348-3e7e8ba84169","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.064187906Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:05.064212714Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.064801546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06494546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06499081Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065159473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065209171Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"31b8587b-02b9-46a0-a348-3e7e8ba84169 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06525948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065297114Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065333466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065372363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065420368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065458573Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.06549272Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"ds_connection connected from Ok(127.0.0.1:34303)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065754317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"ds_connection connected from Ok(127.0.0.1:39040)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065823453Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"ds_connection connected from Ok(127.0.0.1:51298)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.065856478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.067987193Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.068048414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.068219271Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069057545Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069270265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[0]R flush_numbers[0..12]: [292, 294, 294, 286, 277, 294, 285, 279, 288, 288, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069395362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[0]R generation[0..12]: [163, 165, 165, 159, 152, 165, 158, 155, 161, 161, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069433246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069467554Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[1]R flush_numbers[0..12]: [292, 294, 294, 286, 277, 294, 285, 279, 288, 288, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069507964Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[1]R generation[0..12]: [163, 165, 165, 159, 152, 165, 158, 155, 161, 161, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069540428Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069574084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[2]R flush_numbers[0..12]: [292, 294, 294, 286, 277, 294, 285, 279, 288, 288, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069606769Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[2]R generation[0..12]: [163, 165, 165, 159, 152, 165, 158, 155, 161, 161, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069638792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069672609Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069699893Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069728469Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Next flush: 295","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069756064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069783117Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"":"downstairs","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069811703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069837945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"0","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069866251Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"1","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069894807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"client":"2","":"downstairs","session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069923784Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069954746Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"31b8587b-02b9-46a0-a348-3e7e8ba84169 is now active with session: 7cb832d2-cf33-4eb1-ad75-ab96cd681da3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.069987059Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-03-06T06:07:05.070023421Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859,"session_id":"7cb832d2-cf33-4eb1-ad75-ab96cd681da3"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.070052108Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} {"msg":"Activated sub_volume 31b8587b-02b9-46a0-a348-3e7e8ba84169","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:05.070075574Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 502 len 12288 data: 13 12 13 02/30 Flush 03/30 Write block 265 len 8192 data: 11 11 04/30 Read block 54 len 4096 05/30 Write block 427 len 16384 data: 11 8 11 12 06/30 Read block 171 len 8192 07/30 Read block 89 len 16384 08/30 Write block 353 len 32768 data: 13 10 12 15 11 9 12 12 09/30 Read block 121 len 24576 10/30 Write block 414 len 4096 data: 14 11/30 Write block 424 len 24576 data: 15 9 7 12 9 12 12/30 Flush 13/30 Write block 324 len 20480 data: 12 13 10 15 16 14/30 Read block 378 len 8192 15/30 Write block 282 len 20480 data: 19 19 17 17 16 16/30 Read block 442 len 28672 17/30 Write block 4 len 28672 data: 10 9 10 8 12 14 17 18/30 Write block 154 len 20480 data: 10 10 11 12 9 19/30 Flush 20/30 Write block 416 len 24576 data: 13 13 11 12 15 16 21/30 Read block 310 len 32768 22/30 Write block 41 len 24576 data: 7 7 9 9 13 14 23/30 Write block 186 len 24576 data: 15 13 13 12 13 12 24/30 Read block 424 len 8192 25/30 Write block 481 len 16384 data: 11 12 14 22 26/30 Write block 55 len 36864 data: 10 9 9 9 9 9 11 11 12 27/30 Read block 179 len 32768 28/30 Read block 376 len 8192 29/30 Read block 557 len 16384 30/30 Write block 277 len 8192 data: 22 23 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 2 Done Done Done false 1003 Acked Read 1 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1001)) Some(JobId(1018)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:05.096582231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1859} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 166 163 166 297 292 297 F F F <--- 1 020-039 165 165 165 294 294 294 F F F 2 040-059 165 165 165 294 294 294 T F T <--- 3 060-079 159 159 159 286 286 286 T F T <--- 4 080-099 152 152 152 277 277 277 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 166 155 166 297 279 297 F F F <--- 8 160-179 161 161 161 288 288 288 F F F 9 180-199 161 161 161 288 288 288 T F T <--- 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 166 165 166 296 294 296 T T T <--- 14 280-299 166 165 166 297 294 297 F F F <--- 15 300-319 164 164 164 293 293 293 F F F 16 320-339 166 162 166 297 291 297 F F F <--- 17 340-359 166 161 166 296 288 296 F F F <--- 18 360-379 166 162 166 296 289 296 F F F <--- 19 380-399 162 162 162 290 290 290 F F F 20 400-419 166 164 166 296 293 296 T F T <--- 21 420-439 166 165 166 296 294 296 T F T <--- 22 440-459 158 158 158 284 284 284 F F F 23 460-479 165 165 165 294 294 294 F F F 24 480-499 162 162 162 289 289 289 T F T <--- 25 500-519 166 166 166 295 295 295 F F F 26 520-539 164 164 164 293 293 293 F F F 27 540-559 165 165 165 294 294 294 F F F 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 166, Max flush: 297 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 166 166 166 297 297 297 F F F 1 020-039 165 165 165 294 294 294 F F F 2 040-059 167 167 167 298 298 298 F F F 3 060-079 167 167 167 298 298 298 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 158 158 158 285 285 285 F F F 7 140-159 166 166 166 297 297 297 F F F 8 160-179 161 161 161 288 288 288 F F F 9 180-199 167 167 167 298 298 298 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 167 167 167 298 298 298 F F F 14 280-299 166 166 166 297 297 297 F F F 15 300-319 164 164 164 293 293 293 F F F 16 320-339 166 166 166 297 297 297 F F F 17 340-359 166 166 166 296 296 296 F F F 18 360-379 166 166 166 296 296 296 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 167 167 167 298 298 298 F F F 21 420-439 167 167 167 298 298 298 F F F 22 440-459 158 158 158 284 284 284 F F F 23 460-479 165 165 165 294 294 294 F F F 24 480-499 167 167 167 298 298 298 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 164 164 164 293 293 293 F F F 27 540-559 165 165 165 294 294 294 F F F 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 167, Max flush: 298 Begin loop 84 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.649988432Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.671321039Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.672005239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.672098223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.672122671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.67215236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} {"msg":"Crucible 4c37bd8d-afec-4aab-9b99-542f45babe32 has session id: a830f807-e7b7-4ffc-9394-85d8b8544785","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673461523Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Upstairs opts: Upstairs UUID: 4c37bd8d-afec-4aab-9b99-542f45babe32, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673566671Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Crucible stats registered with UUID: 4c37bd8d-afec-4aab-9b99-542f45babe32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673597582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:09.673622801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673680195Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673740003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.673767687Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674344325Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674437269Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} {"msg":"4c37bd8d-afec-4aab-9b99-542f45babe32 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.67448358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674530623Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674582706Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674631412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674679236Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674728273Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674783422Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"ds_connection connected from Ok(127.0.0.1:53027)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674839412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"ds_connection connected from Ok(127.0.0.1:60152)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.674894691Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"ds_connection connected from Ok(127.0.0.1:48361)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.675229412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.67619104Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.676688431Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.677004775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.67734159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.677493901Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[0]R flush_numbers[0..12]: [297, 294, 298, 298, 277, 294, 285, 297, 288, 298, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678262968Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[0]R generation[0..12]: [166, 165, 167, 167, 152, 165, 158, 166, 161, 167, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.67833478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678378356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[1]R flush_numbers[0..12]: [297, 294, 298, 298, 277, 294, 285, 297, 288, 298, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678416922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[1]R generation[0..12]: [166, 165, 167, 167, 152, 165, 158, 166, 161, 167, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678458123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678498443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[2]R flush_numbers[0..12]: [297, 294, 298, 298, 277, 294, 285, 297, 288, 298, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678535586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[2]R generation[0..12]: [166, 165, 167, 167, 152, 165, 158, 166, 161, 167, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678572198Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678610324Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678645022Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678697676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Next flush: 299","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678730561Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678762744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"":"downstairs","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678795639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678826009Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"0","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678939513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"1","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.678995414Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"client":"2","":"downstairs","session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679030302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679060612Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"4c37bd8d-afec-4aab-9b99-542f45babe32 is now active with session: a830f807-e7b7-4ffc-9394-85d8b8544785","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679093377Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679121362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867,"session_id":"a830f807-e7b7-4ffc-9394-85d8b8544785"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679151011Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Activated sub_volume 4c37bd8d-afec-4aab-9b99-542f45babe32","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:09.679191871Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 176 len 32768 data: 10 13 13 12 14 15 15 18 02/30 Read block 499 len 32768 03/30 Write block 21 len 24576 data: 14 10 9 9 9 9 04/30 Read block 460 len 12288 05/30 Read block 106 len 8192 06/30 Read block 197 len 40960 07/30 Write block 479 len 20480 data: 12 12 12 13 15 08/30 Read block 512 len 24576 09/30 Read block 37 len 8192 10/30 Flush 11/30 Write block 531 len 40960 data: 13 13 14 11 11 12 14 14 15 12 12/30 Read block 300 len 16384 13/30 Write block 267 len 20480 data: 11 10 9 8 8 14/30 Read block 305 len 8192 15/30 Read block 113 len 20480 16/30 Write block 146 len 8192 data: 12 10 17/30 Read block 1 len 4096 18/30 Write block 377 len 8192 data: 17 20 19/30 Write block 471 len 4096 data: 17 20/30 Write block 527 len 28672 data: 15 16 14 14 14 14 15 21/30 Write block 125 len 20480 data: 14 13 15 12 9 22/30 Read block 273 len 12288 23/30 Flush 24/30 Flush 25/30 Write block 549 len 24576 data: 16 14 15 16 13 11 26/30 Read block 456 len 8192 27/30 Read block 191 len 16384 28/30 Read block 250 len 4096 29/30 Write block 437 len 36864 data: 16 16 12 12 16 14 15 12 15 30/30 Read block 5 len 4096 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Write 6 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:09.708484276Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1867} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 166 166 166 297 297 297 F F F 1 020-039 168 165 168 299 294 299 F F F <--- 2 040-059 167 167 167 298 298 298 F F F 3 060-079 167 167 167 298 298 298 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 168 158 168 300 285 300 F F F <--- 7 140-159 168 166 168 300 297 300 F F F <--- 8 160-179 168 161 168 299 288 299 F T F <--- 9 180-199 168 167 168 299 298 299 F T F <--- 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 167 168 300 298 300 F F F <--- 14 280-299 166 166 166 297 297 297 F F F 15 300-319 164 164 164 293 293 293 F F F 16 320-339 166 166 166 297 297 297 F F F 17 340-359 166 166 166 296 296 296 F F F 18 360-379 168 166 168 300 296 300 F F F <--- 19 380-399 162 162 162 290 290 290 F F F 20 400-419 167 167 167 298 298 298 F F F 21 420-439 167 167 167 298 298 298 T F T <--- 22 440-459 158 158 158 284 284 284 T F T <--- 23 460-479 168 165 168 300 294 300 F F F <--- 24 480-499 168 167 168 299 298 299 F F F <--- 25 500-519 166 166 166 295 295 295 F F F 26 520-539 168 164 168 300 293 300 F F F <--- 27 540-559 168 165 168 300 294 300 T F T <--- 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 168, Max flush: 300 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 166 166 166 297 297 297 F F F 1 020-039 168 168 168 299 299 299 F F F 2 040-059 167 167 167 298 298 298 F F F 3 060-079 167 167 167 298 298 298 F F F 4 080-099 152 152 152 277 277 277 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 168 168 168 300 300 300 F F F 7 140-159 168 168 168 300 300 300 F F F 8 160-179 168 168 168 299 299 299 F F F 9 180-199 168 168 168 299 299 299 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 168 168 300 300 300 F F F 14 280-299 166 166 166 297 297 297 F F F 15 300-319 164 164 164 293 293 293 F F F 16 320-339 166 166 166 297 297 297 F F F 17 340-359 166 166 166 296 296 296 F F F 18 360-379 168 168 168 300 300 300 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 167 167 167 298 298 298 F F F 21 420-439 169 169 169 301 301 301 F F F 22 440-459 169 169 169 301 301 301 F F F 23 460-479 168 168 168 300 300 300 F F F 24 480-499 168 168 168 299 299 299 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 168 168 168 300 300 300 F F F 27 540-559 169 169 169 301 301 301 F F F 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 169, Max flush: 301 Begin loop 85 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.248862047Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27007069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27056256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.270630154Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.270662618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.270699801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"Crucible d2a39147-c061-4ce9-985d-a309440a5734 has session id: 63286b4c-56a9-43e5-b8e0-1a360963c7d8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.271661579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Upstairs opts: Upstairs UUID: d2a39147-c061-4ce9-985d-a309440a5734, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.271719443Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Crucible stats registered with UUID: d2a39147-c061-4ce9-985d-a309440a5734","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27174832Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:14.271772969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.272274429Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27232578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.272352964Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27296925Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273046743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"d2a39147-c061-4ce9-985d-a309440a5734 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273070419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273095429Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273123354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273150818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273184014Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273211438Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273239012Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:59496)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273540096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:45638)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273803456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:59704)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.273863013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.276530417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.276675403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.276854877Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27728691Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277387378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[0]R flush_numbers[0..12]: [297, 299, 298, 298, 277, 294, 300, 300, 299, 299, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277800403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[0]R generation[0..12]: [166, 168, 167, 167, 152, 165, 168, 168, 168, 168, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277837807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277871163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[1]R flush_numbers[0..12]: [297, 299, 298, 298, 277, 294, 300, 300, 299, 299, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277911092Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[1]R generation[0..12]: [166, 168, 167, 167, 152, 165, 168, 168, 168, 168, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.277942874Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27797624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[2]R flush_numbers[0..12]: [297, 299, 298, 298, 277, 294, 300, 300, 299, 299, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278052942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[2]R generation[0..12]: [166, 168, 167, 167, 152, 165, 168, 168, 168, 168, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278085716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278118912Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278145985Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278174742Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Next flush: 302","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278201304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278227747Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"":"downstairs","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Wait for a query_work_queue command to finish before sending IO No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27826474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278291362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"0","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278319448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"1","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278347232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"client":"2","":"downstairs","session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278375468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27840692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 d2a39147-c061-4ce9-985d-a309440a5734 is now active with session: 63286b4c-56a9-43e5-b8e0-1a360963c7d8","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278440456Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278467329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875,"session_id":"63286b4c-56a9-43e5-b8e0-1a360963c7d8"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.278494413Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} {"msg":"Activated sub_volume d2a39147-c061-4ce9-985d-a309440a5734","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:14.27851841Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 142 len 24576 data: 8 9 12 10 13 11 02/30 Read block 20 len 32768 03/30 Write block 14 len 20480 data: 15 15 15 14 14 04/30 Write block 280 len 36864 data: 20 19 20 20 18 18 17 12 13 05/30 Read block 157 len 32768 06/30 Read block 392 len 24576 07/30 Read block 47 len 20480 08/30 Read block 13 len 16384 09/30 Read block 380 len 12288 10/30 Read block 592 len 20480 11/30 Write block 173 len 4096 data: 11 12/30 Read block 264 len 12288 13/30 Read block 335 len 4096 14/30 Write block 539 len 8192 data: 16 13 15/30 Read block 115 len 28672 16/30 Read block 305 len 4096 17/30 Write block 400 len 28672 data: 19 21 19 22 18 18 15 18/30 Read block 388 len 12288 19/30 Write block 132 len 40960 data: 9 9 7 7 5 9 7 7 8 8 20/30 Flush 21/30 Write block 327 len 28672 data: 16 17 17 18 13 11 11 22/30 Read block 243 len 24576 23/30 Write block 176 len 24576 data: 11 14 14 13 15 16 24/30 Read block 504 len 24576 25/30 Read block 427 len 24576 26/30 Write block 366 len 28672 data: 17 16 12 10 8 9 7 27/30 Read block 68 len 24576 28/30 Write block 91 len 8192 data: 14 15 29/30 Read block 114 len 32768 30/30 Read block 260 len 4096 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Read 8 Done Done Sent false 1002 Acked Write 5 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:14.308615483Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1875} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 166 302 302 297 F F F <--- 1 020-039 168 168 168 299 299 299 F F F 2 040-059 167 167 167 298 298 298 F F F 3 060-079 167 167 167 298 298 298 F F F 4 080-099 152 152 152 277 277 277 T T F <--- 5 100-119 165 165 165 294 294 294 F F F 6 120-139 170 170 168 302 302 300 F F F <--- 7 140-159 170 170 168 302 302 300 F F F <--- 8 160-179 170 170 168 302 302 299 T T F <--- 9 180-199 168 168 168 299 299 299 T T F <--- 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 168 168 300 300 300 F F F 14 280-299 170 170 166 302 302 297 F F F <--- 15 300-319 164 164 164 293 293 293 F F F 16 320-339 166 166 166 297 297 297 T T F <--- 17 340-359 166 166 166 296 296 296 F F F 18 360-379 168 168 168 300 300 300 T T F <--- 19 380-399 162 162 162 290 290 290 F F F 20 400-419 170 170 167 302 302 298 F F F <--- 21 420-439 169 169 169 301 301 301 F F F 22 440-459 169 169 169 301 301 301 F F F 23 460-479 168 168 168 300 300 300 F F F 24 480-499 168 168 168 299 299 299 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 170 170 168 302 302 300 F F F <--- 27 540-559 170 170 169 302 302 301 F F F <--- 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 170, Max flush: 302 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 302 302 302 F F F 1 020-039 168 168 168 299 299 299 F F F 2 040-059 167 167 167 298 298 298 F F F 3 060-079 167 167 167 298 298 298 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 165 165 165 294 294 294 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 170 170 170 302 302 302 F F F 8 160-179 171 171 171 303 303 303 F F F 9 180-199 171 171 171 303 303 303 F F F 10 200-219 161 161 161 288 288 288 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 168 168 300 300 300 F F F 14 280-299 170 170 170 302 302 302 F F F 15 300-319 164 164 164 293 293 293 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 166 166 166 296 296 296 F F F 18 360-379 171 171 171 303 303 303 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 170 170 170 302 302 302 F F F 21 420-439 169 169 169 301 301 301 F F F 22 440-459 169 169 169 301 301 301 F F F 23 460-479 168 168 168 300 300 300 F F F 24 480-499 168 168 168 299 299 299 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 170 170 170 302 302 302 F F F 27 540-559 170 170 170 302 302 302 F F F 28 560-579 164 164 164 293 293 293 F F F 29 580-599 163 163 163 292 292 292 F F F Max gen: 171, Max flush: 303 Begin loop 86 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.849180243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.871039068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.87160231Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.871674743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.871698169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.871732336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"Crucible 66c3215f-fcd2-493b-9f2f-03ad95aa56ab has session id: 22db1f2d-80d6-43d6-ac00-41c9d51fa624","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.872795043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Upstairs opts: Upstairs UUID: 66c3215f-fcd2-493b-9f2f-03ad95aa56ab, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.872853068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Crucible stats registered with UUID: 66c3215f-fcd2-493b-9f2f-03ad95aa56ab","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.872881093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:18.872905151Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873386531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873496898Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873524914Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873824705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873869253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"66c3215f-fcd2-493b-9f2f-03ad95aa56ab active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873966144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.873993749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874026013Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.87405007Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874071332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874102133Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874215787Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"ds_connection connected from Ok(127.0.0.1:54694)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874362707Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"ds_connection connected from Ok(127.0.0.1:47131)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874504397Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"ds_connection connected from Ok(127.0.0.1:58837)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.874777115Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.876594281Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.876782853Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.876823744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877523204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877565347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[0]R flush_numbers[0..12]: [302, 299, 298, 298, 303, 294, 302, 302, 303, 303, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877594996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[0]R generation[0..12]: [170, 168, 167, 167, 171, 165, 170, 170, 171, 171, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877626788Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877659944Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[1]R flush_numbers[0..12]: [302, 299, 298, 298, 303, 294, 302, 302, 303, 303, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877691576Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[1]R generation[0..12]: [170, 168, 167, 167, 171, 165, 170, 170, 171, 171, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877722728Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877755172Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[2]R flush_numbers[0..12]: [302, 299, 298, 298, 303, 294, 302, 302, 303, 303, 288, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877786744Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[2]R generation[0..12]: [170, 168, 167, 167, 171, 165, 170, 170, 171, 171, 161, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877871401Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,Wait for a query_work_queue command to finish before sending IO "":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877951659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.877980947Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.87803806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Next flush: 304","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878065835Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878092347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"":"downstairs","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878120162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878145362Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"0","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878173237Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"1","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878200791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"client":"2","":"downstairs","session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878228806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878255329Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"66c3215f-fcd2-493b-9f2f-03ad95aa56ab is now active with session: 22db1f2d-80d6-43d6-ac00-41c9d51fa624","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878285218Z","hostname":"bmat-EVT22200007-0000ec3aDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878317521Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883,"session_id":"22db1f2d-80d6-43d6-ac00-41c9d51fa624"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878344034Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} {"msg":"Activated sub_volume 66c3215f-fcd2-493b-9f2f-03ad95aa56ab","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:18.878366779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 559 len 4096 data: 13 02/30 Write block 289 len 28672 data: 15 12 13 12 10 11 15 03/30 Write block 344 len 8192 data: 10 9 04/30 Flush 05/30 Write block 161 len 16384 data: 13 17 15 14 06/30 Write block 2 len 12288 data: 8 9 11 07/30 Write block 195 len 4096 data: 11 08/30 Write block 50 len 4096 data: 10 09/30 Read block 434 len 24576 10/30 Flush 11/30 Write block 288 len 12288 data: 14 16 13 12/30 Read block 70 len 36864 13/30 Read block 34 len 36864 14/30 Write block 402 len 36864 data: 20 23 19 19 16 9 11 11 13 15/30 Write block 584 len 12288 data: 11 10 9 16/30 Flush 17/30 Write block 450 len 16384 data: 14 15 15 17 18/30 Write block 54 len 36864 data: 10 11 10 10 10 10 10 12 12 19/30 Read block 561 len 24576 20/30 Write block 411 len 20480 data: 15 16 12 15 12 21/30 Write block 525 len 16384 data: 19 15 16 17 22/30 Write block 203 len 12288 data: 11 12 10 23/30 Write block 288 len 36864 data: 15 17 14 14 13 11 12 16 17 24/30 Write block 280 len 4096 data: 21 25/30 Read block 421 len 8192 26/30 Write block 565 len 24576 data: 14 14 16 15 14 15 27/30 Write block 106 len 8192 data: 11 9 28/30 Read block 370 len 20480 29/30 Write block 306 len 28672 data: 10 11 13 12 11 14 14 30/30 Write block 490 len 32768 data: 10 8 9 10 13 12 12 14 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 2 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 29 2 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:18.902479632Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1883} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 170 172 305 302 305 F F F <--- 1 020-039 168 168 168 299 299 299 F F F 2 040-059 172 167 172 305 298 305 T F T <--- 3 060-079 167 167 167 298 298 298 T F T <--- 4 080-099 171 171 171 303 303 303 F F F 5 100-119 165 165 165 294 294 294 T F T <--- 6 120-139 170 170 170 302 302 302 F F F 7 140-159 170 170 170 302 302 302 F F F 8 160-179 172 171 172 305 303 305 F F F <--- 9 180-199 172 171 172 305 303 305 F F F <--- 10 200-219 161 161 161 288 288 288 T F T <--- 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 168 168 300 300 300 F F F 14 280-299 172 170 172 306 302 306 T T T <--- 15 300-319 164 164 164 293 293 293 T F T <--- 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 166 172 304 296 304 F F F <--- 18 360-379 171 171 171 303 303 303 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 172 170 172 306 302 306 T F T <--- 21 420-439 169 169 169 301 301 301 F F F 22 440-459 169 169 169 301 301 301 T F T <--- 23 460-479 168 168 168 300 300 300 F F F 24 480-499 168 168 168 299 299 299 T F T <--- 25 500-519 166 166 166 295 295 295 F F F 26 520-539 170 170 170 302 302 302 T F T <--- 27 540-559 172 170 172 304 302 304 F T F <--- 28 560-579 164 164 164 293 293 293 T F T <--- 29 580-599 172 163 172 306 292 306 F F F <--- Max gen: 172, Max flush: 306 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 305 305 305 F F F 1 020-039 168 168 168 299 299 299 F F F 2 040-059 173 173 173 307 307 307 F F F 3 060-079 173 173 173 307 307 307 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 173 173 173 307 307 307 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 170 170 170 302 302 302 F F F 8 160-179 172 172 172 305 305 305 F F F 9 180-199 172 172 172 305 305 305 F F F 10 200-219 173 173 173 307 307 307 F F F 11 220-239 164 164 164 293 293 293 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 168 168 168 300 300 300 F F F 14 280-299 173 173 173 307 307 307 F F F 15 300-319 173 173 173 307 307 307 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 F F F 18 360-379 171 171 171 303 303 303 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 173 173 173 307 307 307 F F F 21 420-439 169 169 169 301 301 301 F F F 22 440-459 173 173 173 307 307 307 F F F 23 460-479 168 168 168 300 300 300 F F F 24 480-499 173 173 173 307 307 307 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 173 173 173 307 307 307 F F F 27 540-559 172 172 172 304 304 304 F F F 28 560-579 173 173 173 307 307 307 F F F 29 580-599 172 172 172 306 306 306 F F F Max gen: 173, Max flush: 307 Begin loop 87 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.483562239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.504905387Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.505367339Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.505441695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.505473879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.505508587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"Crucible a06c18e5-a375-4f97-bce8-6388637e20da has session id: d28c41da-2985-49f7-abbb-e069ddd7d1ec","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.506698656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Upstairs opts: Upstairs UUID: a06c18e5-a375-4f97-bce8-6388637e20da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.506757662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Crucible stats registered with UUID: a06c18e5-a375-4f97-bce8-6388637e20da","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.506788233Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:23.506812531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.507188282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.50744542Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.507495899Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508141583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508184658Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"a06c18e5-a375-4f97-bce8-6388637e20da active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508296389Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.50832748Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508354884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508386857Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508412999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508440112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508516513Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:47158)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508949448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:35444)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.508991751Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:38877)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.509020187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.510672418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.510830459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.511577904Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.511822748Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.511855111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[0]R flush_numbers[0..12]: [305, 299, 307, 307, 303, 307, 302, 302, 305, 305, 307, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512658096Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[0]R generation[0..12]: [172, 168, 173, 173, 171, 173, 170, 170, 172, 172, 173, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512697754Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512728886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[1]R flush_numbers[0..12]: [305, 299, 307, 307, 303, 307, 302, 302, 305, 305, 307, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.51276634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[1]R generation[0..12]: [172, 168, 173, 173, 171, 173, 170, 170, 172, 172, 173, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512796299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512826558Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[2]R flush_numbers[0..12]: [305, 299, 307, 307, 303, 307, 302, 302, 305, 305, 307, 293]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512856227Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[2]R generation[0..12]: [172, 168, 173, 173, 171, 173, 170, 170, 172, 172, 173, 164]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512885184Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512915945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512940814Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512967176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Next flush: 308","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.512991805Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513016624Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"":"downstairs","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513043376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513067354Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"0","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO :"2026-03-06T06:07:23.513093676Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"1","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513132162Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"client":"2","":"downstairs","session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513162321Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.5131873Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"a06c18e5-a375-4f97-bce8-6388637e20da is now active with session: d28c41da-2985-49f7-abbb-e069ddd7d1ec","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513215266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513238822Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891,"session_id":"d28c41da-2985-49f7-abbb-e069ddd7d1ec"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513263631Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} {"msg":"Activated sub_volume a06c18e5-a375-4f97-bce8-6388637e20da","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:23.513285274Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 477 len 12288 data: 12 13 13 02/30 Write block 186 len 28672 data: 16 14 14 13 14 13 11 03/30 Write block 418 len 28672 data: 12 13 16 17 16 14 16 04/30 Write block 366 len 16384 data: 18 17 13 11 05/30 Read block 92 len 40960 06/30 Read block 68 len 16384 07/30 Write block 274 len 4096 data: 14 08/30 Read block 2 len 4096 09/30 Read block 274 len 32768 10/30 Read block 327 len 8192 11/30 Write block 574 len 32768 data: 11 10 12 10 9 12 7 8 12/30 Read block 499 len 12288 13/30 Write block 4 len 36864 data: 12 10 11 9 13 15 18 17 16 14/30 Write block 6 len 36864 data: 12 10 14 16 19 18 17 17 16 15/30 Read block 411 len 32768 16/30 Read block 443 len 28672 17/30 Write block 140 len 12288 data: 9 9 9 18/30 Write block 441 len 32768 data: 17 15 16 13 16 15 15 16 19/30 Write block 234 len 8192 data: 11 11 20/30 Read block 146 len 24576 21/30 Flush 22/30 Write block 403 len 12288 data: 24 20 20 23/30 Read block 301 len 32768 24/30 Flush 25/30 Write block 310 len 24576 data: 12 15 15 15 13 17 26/30 Read block 573 len 4096 27/30 Write block 15 len 20480 data: 16 16 15 15 12 28/30 Write block 460 len 20480 data: 12 12 12 15 13 29/30 Read block 476 len 36864 30/30 Write block 521 len 28672 data: 16 15 14 15 20 16 17 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Write 7 Done Sent Done false 1002 Acked Write 7 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:23.540700474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1891} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 174 172 174 308 305 308 T F T <--- 1 020-039 168 168 168 299 299 299 F F F 2 040-059 173 173 173 307 307 307 F F F 3 060-079 173 173 173 307 307 307 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 173 173 173 307 307 307 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 174 170 174 308 302 308 F F F <--- 8 160-179 172 172 172 305 305 305 F F F 9 180-199 174 172 174 308 305 308 F F F <--- 10 200-219 173 173 173 307 307 307 F F F 11 220-239 174 164 174 308 293 308 F F F <--- 12 240-259 164 164 164 293 293 293 F F F 13 260-279 174 168 174 308 300 308 F F F <--- 14 280-299 173 173 173 307 307 307 F F F 15 300-319 173 173 173 307 307 307 T F T <--- 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 F F F 18 360-379 174 171 174 308 303 308 F F F <--- 19 380-399 162 162 162 290 290 290 F F F 20 400-419 174 173 174 309 307 309 F F F <--- 21 420-439 174 169 174 308 301 308 F F F <--- 22 440-459 174 173 174 308 307 308 F F F <--- 23 460-479 174 168 174 308 300 308 T F T <--- 24 480-499 173 173 173 307 307 307 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 173 173 173 307 307 307 T F T <--- 27 540-559 172 172 172 304 304 304 F F F 28 560-579 174 173 174 308 307 308 F F F <--- 29 580-599 174 172 174 308 306 308 F F F <--- Max gen: 174, Max flush: 309 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 310 310 310 F F F 1 020-039 168 168 168 299 299 299 F F F 2 040-059 173 173 173 307 307 307 F F F 3 060-079 173 173 173 307 307 307 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 173 173 173 307 307 307 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 174 174 174 308 308 308 F F F 8 160-179 172 172 172 305 305 305 F F F 9 180-199 174 174 174 308 308 308 F F F 10 200-219 173 173 173 307 307 307 F F F 11 220-239 174 174 174 308 308 308 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 174 174 174 308 308 308 F F F 14 280-299 173 173 173 307 307 307 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 F F F 18 360-379 174 174 174 308 308 308 F F F 19 380-399 162 162 162 290 290 290 F F F 20 400-419 174 174 174 309 309 309 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 174 174 174 308 308 308 F F F 23 460-479 175 175 175 310 310 310 F F F 24 480-499 173 173 173 307 307 307 F F F 25 500-519 166 166 166 295 295 295 F F F 26 520-539 175 175 175 310 310 310 F F F 27 540-559 172 172 172 304 304 304 F F F 28 560-579 174 174 174 308 308 308 F F F 29 580-599 174 174 174 308 308 308 F F F Max gen: 175, Max flush: 310 Begin loop 88 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.081457302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.101994199Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.102522201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.102654101Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.10269929Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.102751634Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"Crucible 5eaacab6-040b-4d26-838e-73a26d0e6c14 has session id: 01835a2b-73d6-4dbb-b608-fd276c393d57","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.103912084Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Upstairs opts: Upstairs UUID: 5eaacab6-040b-4d26-838e-73a26d0e6c14, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.103973575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Crucible stats registered with UUID: 5eaacab6-040b-4d26-838e-73a26d0e6c14","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.104001971Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:28.104032702Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.104409435Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.104462149Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.104551235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.104798232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105005051Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"5eaacab6-040b-4d26-838e-73a26d0e6c14 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.10508069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105110298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.10514138Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105170216Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105378908Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105480799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105523203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"ds_connection connected from Ok(127.0.0.1:47223)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105562972Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"ds_connection connected from Ok(127.0.0.1:52071)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105601888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"ds_connection connected from Ok(127.0.0.1:44061)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.105640825Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.107767291Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.107815316Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.107988077Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.108808646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.108887322Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[0]R flush_numbers[0..12]: [310, 299, 307, 307, 303, 307, 302, 308, 305, 308, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.10891686Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[0]R generation[0..12]: [175, 168, 173, 173, 171, 173, 170, 174, 172, 174, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.108948412Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.108979534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[1]R flush_numbers[0..12]: [310, 299, 307, 307, 303, 307, 302, 308, 305, 308, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109009352Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[1]R generation[0..12]: [175, 168, 173, 173, 171, 173, 170, 174, 172, 174, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109039382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109069852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[2]R flush_numbers[0..12]: [310, 299, 307, 307, 303, 307, 302, 308, 305, 308, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.10909907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[2]R generation[0..12]: [175, 168, 173, 173, 171, 173, 170, 174, 172, 174, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109128578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109158958Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109193486Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109220459Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Next flush: 311","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109245448Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109270518Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"":"downstairs","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109296299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109328703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"0","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109356457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"1","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109382789Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"client":"2","":"downstairs","session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109409141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"5eaacab6-040b-4d26-838e-73a26d0e6c14 is now active with session: 01835a2b-73d6-4dbb-b608-fd276c393d57","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109461966Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109485382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899,"session_id":"01835a2b-73d6-4dbb-b608-fd276c393d57"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.1095099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} {"msg":"Activated sub_volume 5eaacab6-040b-4d26-838e-73a26d0e6c14","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:28.109531483Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 11 len 24576 data: 19 18 18 17 17 17 02/30 Flush 03/30 Write block 379 len 40960 data: 23 22 25 21 20 19 17 15 14 13 04/30 Write block 194 len 20480 data: 11 12 12 13 13 05/30 Write block 7 len 24576 data: 11 15 17 20 20 19 06/30 Write block 25 len 40960 data: 10 10 13 16 20 19 14 16 14 15 07/30 Flush 08/30 Write block 579 len 32768 data: 13 8 9 9 11 12 11 10 09/30 Flush 10/30 Write block 455 len 4096 data: 14 11/30 Read block 324 len 32768 12/30 Read block 97 len 40960 13/30 Read block 324 len 40960 14/30 Write block 45 len 32768 data: 14 15 10 11 9 11 7 5 15/30 Read block 451 len 20480 16/30 Write block 478 len 20480 data: 14 14 13 13 14 17/30 Read block 448 len 32768 18/30 Read block 460 len 40960 19/30 Write block 270 len 36864 data: 9 9 12 16 15 20 22 23 24 20/30 Read block 403 len 4096 21/30 Write block 405 len 12288 data: 21 17 10 22/30 Read block 404 len 20480 23/30 Write block 498 len 12288 data: 15 13 13 24/30 Write block 149 len 4096 data: 9 25/30 Flush 26/30 Write block 22 len 12288 data: 11 10 10 27/30 Read block 62 len 24576 28/30 Write block 575 len 20480 data: 11 13 11 10 14 29/30 Write block 42 len 36864 data: 8 10 10 15 16 11 12 10 12 30/30 Read block 209 len 16384 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 10 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 6 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 0 24 Done 28 4 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1001)) Some(JobId(1024)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:28.140275813Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1899} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 311 312 F T F <--- 1 020-039 176 168 176 312 299 312 T T T <--- 2 040-059 176 173 176 314 307 314 T F T <--- 3 060-079 173 173 173 307 307 307 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 173 173 173 307 307 307 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 176 174 176 314 308 314 F F F <--- 8 160-179 172 172 172 305 305 305 F F F 9 180-199 176 174 176 312 308 312 F T F <--- 10 200-219 173 173 173 307 307 307 F F F 11 220-239 174 174 174 308 308 308 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 174 176 314 308 314 F F F <--- 14 280-299 173 173 173 307 307 307 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 F F F 18 360-379 176 174 176 312 308 312 F T F <--- 19 380-399 176 162 176 312 290 312 F T F <--- 20 400-419 176 174 176 314 309 314 F F F <--- 21 420-439 174 174 174 308 308 308 F F F 22 440-459 176 174 176 314 308 314 F F F <--- 23 460-479 176 175 176 314 310 314 F F F <--- 24 480-499 176 173 176 314 307 314 F F F <--- 25 500-519 176 166 176 314 295 314 F F F <--- 26 520-539 175 175 175 310 310 310 F F F 27 540-559 172 172 172 304 304 304 F F F 28 560-579 176 174 176 313 308 313 T F T <--- 29 580-599 176 174 176 313 308 313 F F F <--- Max gen: 176, Max flush: 314 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F F F 1 020-039 177 177 177 315 315 315 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 173 173 173 307 307 307 F F F 4 080-099 171 171 171 303 303 303 F F F 5 100-119 173 173 173 307 307 307 F F F 6 120-139 170 170 170 302 302 302 F F F 7 140-159 176 176 176 314 314 314 F F F 8 160-179 172 172 172 305 305 305 F F F 9 180-199 176 176 176 312 312 312 F F F 10 200-219 173 173 173 307 307 307 F F F 11 220-239 174 174 174 308 308 308 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 176 176 314 314 314 F F F 14 280-299 173 173 173 307 307 307 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 F F F 18 360-379 176 176 176 312 312 312 F F F 19 380-399 176 176 176 312 312 312 F F F 20 400-419 176 176 176 314 314 314 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 176 176 176 314 314 314 F F F 23 460-479 176 176 176 314 314 314 F F F 24 480-499 176 176 176 314 314 314 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 175 175 175 310 310 310 F F F 27 540-559 172 172 172 304 304 304 F F F 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F F F Max gen: 177, Max flush: 315 Begin loop 89 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.697745922Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.718788738Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.719274105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.719353973Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.719378642Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.719412649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"Crucible 20d42a31-70c8-4c33-846e-dcfb1169f5d3 has session id: 43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720401109Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Upstairs opts: Upstairs UUID: 20d42a31-70c8-4c33-846e-dcfb1169f5d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720460867Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Crucible stats registered with UUID: 20d42a31-70c8-4c33-846e-dcfb1169f5d3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720489524Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:32.720514152Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720870446Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720918981Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.720946115Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.721444629Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.721479097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"20d42a31-70c8-4c33-846e-dcfb1169f5d3 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.721785712Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.721814088Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.72184578Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"1","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.721871792Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.722001528Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.72203317Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.722059201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:53850)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.722421446Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:36579)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.722465343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:38048)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.722493759Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.72433881Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"1","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.724577731Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.724653781Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725179409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725294175Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[0]R flush_numbers[0..12]: [312, 315, 315, 307, 303, 307, 302, 314, 305, 312, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.72544841Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[0]R generation[0..12]: [176, 177, 177, 173, 171, 173, 170, 176, 172, 176, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725484962Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725527646Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[1]R flush_numbers[0..12]: [312, 315, 315, 307, 303, 307, 302, 314, 305, 312, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.72555993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[1]R generation[0..12]: [176, 177, 177, 173, 171, 173, 170, 176, 172, 176, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725591953Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725625208Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[2]R flush_numbers[0..12]: [312, 315, 315, 307, 303, 307, 302, 314, 305, 312, 307, 308]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725657121Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[2]R generation[0..12]: [176, 177, 177, 173, 171, 173, 170, 176, 172, 176, 173, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725688373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725721238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725748482Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725776327Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Next flush: 316","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725802849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725829482Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"":"downstairs","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725857847Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725883408Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"0","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725911363Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"1","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725948536Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"client":"2","":"downstairs","session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.725981562Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.726008575Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"20d42a31-70c8-4c33-846e-dcfb1169f5d3 is now active with session: 43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.726038865Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.726064235Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907,"session_id":"43084cc7-c9e8-4b9d-b675-9cfa0d6ba6b2"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.726091228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} {"msg":"Activated sub_volume 20d42a31-70c8-4c33-846e-dcfb1169f5d3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:32.726114374Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 108 len 8192 data: 10 12 02/30 Write block 187 len 8192 data: 15 15 03/30 Write block 122 len 40960 data: 14 15 17 15 14 16 13 10 14 11 04/30 Write block 351 len 12288 data: 12 12 14 05/30 Write block 194 len 20480 data: 12 13 13 14 14 06/30 Read block 492 len 24576 07/30 Write block 73 len 36864 data: 13 15 17 18 19 18 16 18 21 08/30 Write block 546 len 4096 data: 11 09/30 Write block 557 len 12288 data: 13 13 14 10/30 Write block 216 len 40960 data: 6 11 13 16 15 14 15 15 13 14 11/30 Read block 294 len 32768 12/30 Read block 217 len 24576 13/30 Read block 495 len 12288 14/30 Read block 337 len 40960 15/30 Write block 33 len 20480 data: 15 16 16 10 15 16/30 Read block 424 len 32768 17/30 Write block 194 len 36864 data: 13 14 14 15 15 11 11 10 10 18/30 Read block 112 len 8192 19/30 Read block 349 len 24576 20/30 Read block 352 len 36864 21/30 Write block 156 len 8192 data: 12 13 22/30 Read block 96 len 12288 23/30 Read block 312 len 24576 24/30 Read block 409 len 4096 25/30 Read block 427 len 20480 26/30 Read block 480 len 28672 27/30 Write block 545 len 40960 data: 10 12 13 12 17 15 16 17 14 12 28/30 Read block 109 len 28672 29/30 Read block 74 len 40960 30/30 Read block 231 len 24576 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Write 9 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Write 3 Done Done Done false 1009 Acked Write 10 Done Done Done false 1010 Acked Read 8 Done Done Done false 1011 Acked Read 6 Done Done Done false 1012 Acked Read 3 Done Done Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 17 0 17 Done 30 13 30 73 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:32.756416852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1907} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F F F 1 020-039 177 177 177 315 315 315 T F T <--- 2 040-059 177 177 177 315 315 315 F F F 3 060-079 173 173 173 307 307 307 T T T 4 080-099 171 171 171 303 303 303 T T T 5 100-119 173 173 173 307 307 307 T T T 6 120-139 170 170 170 302 302 302 T T T 7 140-159 176 176 176 314 314 314 T F T <--- 8 160-179 172 172 172 305 305 305 F F F 9 180-199 176 176 176 312 312 312 T T T 10 200-219 173 173 173 307 307 307 T T T 11 220-239 174 174 174 308 308 308 T T T 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 176 176 314 314 314 F F F 14 280-299 173 173 173 307 307 307 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 172 172 172 304 304 304 T T T 18 360-379 176 176 176 312 312 312 F F F 19 380-399 176 176 176 312 312 312 F F F 20 400-419 176 176 176 314 314 314 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 176 176 176 314 314 314 F F F 23 460-479 176 176 176 314 314 314 F F F 24 480-499 176 176 176 314 314 314 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 175 175 175 310 310 310 F F F 27 540-559 172 172 172 304 304 304 T T T 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F F F Max gen: 177, Max flush: 315 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F F F 1 020-039 178 178 178 316 316 316 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 178 178 178 316 316 316 F F F 4 080-099 178 178 178 316 316 316 F F F 5 100-119 178 178 178 316 316 316 F F F 6 120-139 178 178 178 316 316 316 F F F 7 140-159 178 178 178 316 316 316 F F F 8 160-179 172 172 172 305 305 305 F F F 9 180-199 178 178 178 316 316 316 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 178 178 178 316 316 316 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 176 176 314 314 314 F F F 14 280-299 173 173 173 307 307 307 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 178 178 178 316 316 316 F F F 18 360-379 176 176 176 312 312 312 F F F 19 380-399 176 176 176 312 312 312 F F F 20 400-419 176 176 176 314 314 314 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 176 176 176 314 314 314 F F F 23 460-479 176 176 176 314 314 314 F F F 24 480-499 176 176 176 314 314 314 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 175 175 175 310 310 310 F F F 27 540-559 178 178 178 316 316 316 F F F 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F F F Max gen: 178, Max flush: 316 Begin loop 90 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.290469992Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:37.311882356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.311967063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.334381265Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.334831553Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.334896551Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.334919427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.334953153Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"Crucible ad9ef7ed-a653-4822-a9c6-d970973b26dd has session id: 8c068b07-7bdf-429a-ada6-1fd1177a0d74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336031822Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Upstairs opts: Upstairs UUID: ad9ef7ed-a653-4822-a9c6-d970973b26dd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336099305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Crucible stats registered with UUID: ad9ef7ed-a653-4822-a9c6-d970973b26dd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336145446Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:37.336178471Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336497511Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.336812082Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337018288Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337046454Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"ad9ef7ed-a653-4822-a9c6-d970973b26dd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337299543Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337487965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.33762703Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337665125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.33769306Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337722017Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.337977541Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"ds_connection connected from Ok(127.0.0.1:55711)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.338024214Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"ds_connection connected from Ok(127.0.0.1:48147)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.33807833Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"ds_connection connected from Ok(127.0.0.1:64669)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.338115464Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.340034791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.340255567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.34030285Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.34114931Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341204018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[0]R flush_numbers[0..12]: [312, 316, 315, 316, 316, 316, 316, 316, 305, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341247043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[0]R generation[0..12]: [176, 178, 177, 178, 178, 178, 178, 178, 172, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341292733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341340247Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[1]R flush_numbers[0..12]: [312, 316, 315, 316, 316, 316, 316, 316, 305, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341377791Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[1]R generation[0..12]: [176, 178, 177, 178, 178, 178, 178, 178, 172, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341415065Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341462919Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R flush_numbers[0..12]: [312, 316, 315, 316, 316, 316, 316, 316, 305, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341571884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[2]R generation[0..12]: [176, 178, 177, 178, 178, 178, 178, 178, 172, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341791628Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341852608Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Max found gen is 179","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341878018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Generation requested: 180 >= found:179","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341902928Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341925743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341948868Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"":"downstairs","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.341972866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.34199553Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"0","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.34202051Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"1","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342053755Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"client":"2","":"downstairs","session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342079897Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342103584Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"ad9ef7ed-a653-4822-a9c6-d970973b26dd is now active with session: 8c068b07-7bdf-429a-ada6-1fd1177a0d74","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342129405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342151328Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915,"session_id":"8c068b07-7bdf-429a-ada6-1fd1177a0d74"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342174223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} {"msg":"Activated sub_volume ad9ef7ed-a653-4822-a9c6-d970973b26dd","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:37.342194112Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 443 len 16384 data: 17 14 17 16 03/30 Write block 293 len 20480 data: 12 13 17 18 20 04/30 Write block 414 len 4096 data: 16 05/30 Read block 410 len 24576 06/30 Read block 348 len 36864 07/30 Read block 372 len 16384 08/30 Write block 367 len 32768 data: 18 14 12 9 10 8 11 13 09/30 Read block 205 len 36864 10/30 Read block 506 len 8192 11/30 Read block 87 len 20480 12/30 Write block 75 len 16384 data: 18 19 20 19 13/30 Write block 98 len 24576 data: 6 7 9 11 11 12 14/30 Read block 118 len 40960 15/30 Read block 3 len 36864 16/30 Write block 489 len 40960 data: 12 11 9 10 11 14 13 13 15 16 17/30 Write block 474 len 36864 data: 16 16 13 13 15 15 14 14 15 18/30 Read block 142 len 20480 19/30 Write block 524 len 36864 data: 16 21 17 18 18 15 15 15 15 20/30 Write block 280 len 16384 data: 22 20 21 21 21/30 Flush 22/30 Flush 23/30 Write block 193 len 24576 data: 9 14 15 15 16 16 24/30 Read block 71 len 4096 25/30 Read block 103 len 4096 26/30 Read block 575 len 4096 27/30 Write block 161 len 4096 data: 14 28/30 Read block 527 len 36864 29/30 Read block 434 len 28672 30/30 Write block 553 len 28672 data: 15 13 13 13 14 14 15 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:37.372979824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1915} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F F F 1 020-039 178 178 178 316 316 316 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 178 180 180 316 318 318 F F F <--- 4 080-099 178 180 180 316 318 318 F F F <--- 5 100-119 178 180 180 316 318 318 F F F <--- 6 120-139 178 178 178 316 316 316 F F F 7 140-159 178 178 178 316 316 316 F F F 8 160-179 172 172 172 305 305 305 F T T <--- 9 180-199 178 178 178 316 316 316 F T T <--- 10 200-219 178 178 178 316 316 316 F F F 11 220-239 178 178 178 316 316 316 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 176 176 314 314 314 F F F 14 280-299 173 180 180 307 318 318 F F F <--- 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 178 178 178 316 316 316 F F F 18 360-379 176 180 180 312 318 318 F F F <--- 19 380-399 176 176 176 312 312 312 F F F 20 400-419 176 180 180 314 318 318 F F F <--- 21 420-439 174 174 174 308 308 308 F F F 22 440-459 176 180 180 314 318 318 F F F <--- 23 460-479 176 180 180 314 318 318 F F F <--- 24 480-499 176 180 180 314 318 318 F F F <--- 25 500-519 176 176 176 314 314 314 F F F 26 520-539 175 180 180 310 318 318 F F F <--- 27 540-559 178 178 178 316 316 316 F T T <--- 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F F F Max gen: 180, Max flush: 318 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F F F 1 020-039 178 178 178 316 316 316 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 180 180 180 318 318 318 F F F 4 080-099 180 180 180 318 318 318 F F F 5 100-119 180 180 180 318 318 318 F F F 6 120-139 178 178 178 316 316 316 F F F 7 140-159 178 178 178 316 316 316 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 178 178 178 316 316 316 F F F 12 240-259 164 164 164 293 293 293 F F F 13 260-279 176 176 176 314 314 314 F F F 14 280-299 180 180 180 318 318 318 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 171 171 303 303 303 F F F 17 340-359 178 178 178 316 316 316 F F F 18 360-379 180 180 180 318 318 318 F F F 19 380-399 176 176 176 312 312 312 F F F 20 400-419 180 180 180 318 318 318 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 180 180 180 318 318 318 F F F 23 460-479 180 180 180 318 318 318 F F F 24 480-499 180 180 180 318 318 318 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 180 180 180 318 318 318 F F F 27 540-559 181 181 181 319 319 319 F F F 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F F F Max gen: 181, Max flush: 319 Begin loop 91 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.924960772Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:41.944857656Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.944923335Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.962036927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.962469009Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.962554458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.962590509Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.962640979Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"Crucible c16a6773-1f76-4fcd-9760-138f24c2002b has session id: 4b1e057c-bc9c-438a-9868-e32c5cc1f219","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.96380712Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Upstairs opts: Upstairs UUID: c16a6773-1f76-4fcd-9760-138f24c2002b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.963880916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Crucible stats registered with UUID: c16a6773-1f76-4fcd-9760-138f24c2002b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.963920534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:41.96395415Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.964461732Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.964611918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.964652879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.964886029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.964928733Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"c16a6773-1f76-4fcd-9760-138f24c2002b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.96511457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.96514998Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965186492Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965219818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"2","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965249887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965286409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965536212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"ds_connection connected from Ok(127.0.0.1:38543)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.965958405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"ds_connection connected from Ok(127.0.0.1:51844)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.966039866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"ds_connection connected from Ok(127.0.0.1:56442)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.966087199Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.967497172Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"2","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.967551749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.967588592Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.96832203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968455533Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[0]R flush_numbers[0..12]: [312, 316, 315, 318, 318, 318, 316, 316, 319, 319, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968607403Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[0]R generation[0..12]: [176, 178, 177, 180, 180, 180, 178, 178, 181, 181, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968650087Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.96868695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[1]R flush_numbers[0..12]: [312, 316, 315, 318, 318, 318, 316, 316, 319, 319, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968721959Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[1]R generation[0..12]: [176, 178, 177, 180, 180, 180, 178, 178, 181, 181, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968756777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968793029Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[2]R flush_numbers[0..12]: [312, 316, 315, 318, 318, 318, 316, 316, 319, 319, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968828018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[2]R generation[0..12]: [176, 178, 177, 180, 180, 180, 178, 178, 181, 181, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968862807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968905852Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968938095Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.968969277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Next flush: 320","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969021099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969050878Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"":"downstairs","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969087069Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969115495Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"0","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969146376Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"1","":"downstairs","session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969195173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"client":"2","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969248859Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969295831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"c16a6773-1f76-4fcd-9760-138f24c2002b is now active with session: 4b1e057c-bc9c-438a-9868-e32c5cc1f219","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969337163Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969370278Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923,"session_id":"4b1e057c-bc9c-438a-9868-e32c5cc1f219"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969412201Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} {"msg":"Activated sub_volume c16a6773-1f76-4fcd-9760-138f24c2002b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:41.969451779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 377 len 28672 data: 18 21 24 23 26 22 21 03/30 Write block 326 len 20480 data: 11 17 18 18 19 04/30 Write block 234 len 8192 data: 12 12 05/30 Write block 357 len 32768 data: 12 10 13 13 13 13 15 18 06/30 Flush 07/30 Write block 295 len 8192 data: 18 19 08/30 Read block 489 len 40960 09/30 Write block 558 len 8192 data: 15 16 10/30 Flush 11/30 Write block 332 len 8192 data: 12 12 12/30 Write block 129 len 40960 data: 11 15 12 10 10 8 8 6 10 8 13/30 Read block 469 len 32768 14/30 Read block 62 len 28672 15/30 Write block 36 len 8192 data: 11 16 16/30 Flush 17/30 Write block 271 len 16384 data: 10 13 17 16 18/30 Write block 258 len 4096 data: 15 19/30 Write block 582 len 12288 data: 10 12 13 20/30 Read block 254 len 36864 21/30 Read block 572 len 32768 22/30 Write block 29 len 40960 data: 21 20 15 17 16 17 17 12 17 15 23/30 Read block 469 len 36864 24/30 Read block 63 len 24576 25/30 Write block 17 len 24576 data: 16 16 13 13 15 12 26/30 Write block 322 len 24576 data: 14 13 13 14 12 18 27/30 Read block 216 len 8192 28/30 Read block 265 len 16384 29/30 Write block 476 len 24576 data: 14 14 16 16 15 15 30/30 Read block 226 len 20480 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 0 27 Done 3 28 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:41.998111705Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1923} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 312 312 312 F T T <--- 1 020-039 178 182 182 316 323 323 F T T <--- 2 040-059 177 177 177 315 315 315 F F F 3 060-079 180 180 180 318 318 318 F F F 4 080-099 180 180 180 318 318 318 F F F 5 100-119 180 180 180 318 318 318 F F F 6 120-139 178 182 182 316 323 323 F F F <--- 7 140-159 178 178 178 316 316 316 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 178 182 182 316 321 321 T F F <--- 12 240-259 164 164 164 293 293 293 F T T <--- 13 260-279 176 176 176 314 314 314 F T T <--- 14 280-299 180 182 182 318 322 322 F F F <--- 15 300-319 175 175 175 310 310 310 F F F 16 320-339 171 182 182 303 323 323 T T T <--- 17 340-359 178 182 182 316 321 321 F F F <--- 18 360-379 180 182 182 318 321 321 T F F <--- 19 380-399 176 182 182 312 321 321 T F F <--- 20 400-419 180 180 180 318 318 318 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 180 180 180 318 318 318 F F F 23 460-479 180 180 180 318 318 318 F T T <--- 24 480-499 180 180 180 318 318 318 F T T <--- 25 500-519 176 176 176 314 314 314 F F F 26 520-539 180 180 180 318 318 318 F F F 27 540-559 181 182 182 319 322 322 F F F <--- 28 560-579 177 177 177 315 315 315 F F F 29 580-599 176 176 176 313 313 313 F T T <--- Max gen: 182, Max flush: 323 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 324 324 324 F F F 1 020-039 183 183 183 324 324 324 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 180 180 180 318 318 318 F F F 4 080-099 180 180 180 318 318 318 F F F 5 100-119 180 180 180 318 318 318 F F F 6 120-139 182 182 182 323 323 323 F F F 7 140-159 178 178 178 316 316 316 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 182 182 182 321 321 321 F F F 12 240-259 183 183 183 324 324 324 F F F 13 260-279 183 183 183 324 324 324 F F F 14 280-299 182 182 182 322 322 322 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 183 183 183 324 324 324 F F F 17 340-359 182 182 182 321 321 321 F F F 18 360-379 182 182 182 321 321 321 F F F 19 380-399 182 182 182 321 321 321 F F F 20 400-419 180 180 180 318 318 318 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 180 180 180 318 318 318 F F F 23 460-479 183 183 183 324 324 324 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 180 180 180 318 318 318 F F F 27 540-559 182 182 182 322 322 322 F F F 28 560-579 177 177 177 315 315 315 F F F 29 580-599 183 183 183 324 324 324 F F F Max gen: 183, Max flush: 324 Begin loop 92 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.592318721Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.617592907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.61822307Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.618320382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.618348337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.618389058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"Crucible fdfdeeff-1f34-4889-9f3f-75248f4039e0 has session id: 6a9ff144-1108-4f2b-83a5-7b7bad26520e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.619451915Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Upstairs opts: Upstairs UUID: fdfdeeff-1f34-4889-9f3f-75248f4039e0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.619521041Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Crucible stats registered with UUID: fdfdeeff-1f34-4889-9f3f-75248f4039e0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.619555449Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:46.619584497Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620281232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620387782Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.62042258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620454203Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620480976Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"fdfdeeff-1f34-4889-9f3f-75248f4039e0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620664187Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620699427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620730588Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620761579Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620792801Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620825145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.620968979Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44061)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.621355311Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61178)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.621465969Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"ds_connection connected from Ok(127.0.0.1:62721)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.621514385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.623127398Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.62334129Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.623390457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624154064Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624373186Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[0]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 316, 319, 319, 316, 321]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624431411Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[0]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 178, 181, 181, 178, 182]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624470488Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624515167Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[1]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 316, 319, 319, 316, 321]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624554174Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[1]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 178, 181, 181, 178, 182]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.62459276Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624640093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[2]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 316, 319, 319, 316, 321]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624685012Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[2]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 178, 181, 181, 178, 182]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624720141Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624756383Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624794418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624827964Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Next flush: 325","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624857462Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.62488692Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"":"downstairs","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624918262Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624946368Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"0","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.624977589Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"1","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.62501361Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"client":"2","":"downstairs","session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"All required reconciliation work is completed","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-06T06:07:46.625070743Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.625112916Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"fdfdeeff-1f34-4889-9f3f-75248f4039e0 is now active with session: 6a9ff144-1108-4f2b-83a5-7b7bad26520e","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.625156372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.625190479Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931,"session_id":"6a9ff144-1108-4f2b-83a5-7b7bad26520e"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.625224266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} {"msg":"Activated sub_volume fdfdeeff-1f34-4889-9f3f-75248f4039e0","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:46.625250458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 557 len 24576 data: 15 16 17 10 9 10 02/30 Flush 03/30 Write block 452 len 16384 data: 16 18 15 15 04/30 Read block 23 len 28672 05/30 Read block 457 len 16384 06/30 Read block 144 len 4096 07/30 Flush 08/30 Write block 149 len 8192 data: 10 12 09/30 Read block 19 len 28672 10/30 Read block 325 len 36864 11/30 Write block 585 len 16384 data: 12 11 11 10 12/30 Read block 440 len 24576 13/30 Read block 21 len 4096 14/30 Write block 463 len 16384 data: 16 14 16 15 15/30 Write block 446 len 16384 data: 17 16 17 14 16/30 Write block 557 len 16384 data: 16 17 18 11 17/30 Read block 438 len 4096 18/30 Write block 227 len 24576 data: 13 13 11 11 12 12 19/30 Read block 498 len 16384 20/30 Read block 351 len 8192 21/30 Read block 291 len 32768 22/30 Read block 504 len 12288 23/30 Read block 388 len 32768 24/30 Flush 25/30 Write block 522 len 32768 data: 16 15 17 22 18 19 19 16 26/30 Write block 273 len 36864 data: 18 17 21 23 24 25 19 23 21 27/30 Write block 532 len 20480 data: 16 16 12 12 13 28/30 Write block 525 len 16384 data: 23 19 20 20 29/30 Read block 429 len 28672 30/30 Write block 148 len 16384 data: 12 11 13 13 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:46.653948088Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1931} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 324 324 324 F F F 1 020-039 183 183 183 324 324 324 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 180 180 180 318 318 318 F F F 4 080-099 180 180 180 318 318 318 F F F 5 100-119 180 180 180 318 318 318 F F F 6 120-139 182 182 182 323 323 323 F F F 7 140-159 184 184 178 327 327 316 T T F <--- 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 184 184 182 327 327 321 F F F <--- 12 240-259 183 183 183 324 324 324 F F F 13 260-279 183 183 183 324 324 324 T T F <--- 14 280-299 182 182 182 322 322 322 T T F <--- 15 300-319 175 175 175 310 310 310 F F F 16 320-339 183 183 183 324 324 324 F F F 17 340-359 182 182 182 321 321 321 F F F 18 360-379 182 182 182 321 321 321 F F F 19 380-399 182 182 182 321 321 321 F F F 20 400-419 180 180 180 318 318 318 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 184 184 180 327 327 318 F F F <--- 23 460-479 184 184 183 327 327 324 F F F <--- 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 180 180 180 318 318 318 T T F <--- 27 540-559 184 184 182 327 327 322 F F T <--- 28 560-579 184 184 177 327 327 315 F F T <--- 29 580-599 184 184 183 327 327 324 F F F <--- Max gen: 184, Max flush: 327 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 324 324 324 F F F 1 020-039 183 183 183 324 324 324 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 180 180 180 318 318 318 F F F 4 080-099 180 180 180 318 318 318 F F F 5 100-119 180 180 180 318 318 318 F F F 6 120-139 182 182 182 323 323 323 F F F 7 140-159 185 185 185 328 328 328 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 184 184 184 327 327 327 F F F 12 240-259 183 183 183 324 324 324 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 185 185 185 328 328 328 F F F 15 300-319 175 175 175 310 310 310 F F F 16 320-339 183 183 183 324 324 324 F F F 17 340-359 182 182 182 321 321 321 F F F 18 360-379 182 182 182 321 321 321 F F F 19 380-399 182 182 182 321 321 321 F F F 20 400-419 180 180 180 318 318 318 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 184 184 184 327 327 327 F F F 23 460-479 184 184 184 327 327 327 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 185 185 185 328 328 328 F F F 27 540-559 184 184 184 327 327 327 F F F 28 560-579 184 184 184 327 327 327 F F F 29 580-599 184 184 184 327 327 327 F F F Max gen: 185, Max flush: 328 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.189922189Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.211291418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21181963Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.211943505Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21198169Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.212023372Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"Crucible 58700fc4-1b3d-406a-aa5b-efcf30e039d4 has session id: c687f4ff-be90-458c-97e4-4ab20bf937a5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.212885564Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Upstairs opts: Upstairs UUID: 58700fc4-1b3d-406a-aa5b-efcf30e039d4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.212963277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Crucible stats registered with UUID: 58700fc4-1b3d-406a-aa5b-efcf30e039d4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.212997344Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:51.213026803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.213398957Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.213580585Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.213622147Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21402391Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21413531Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"58700fc4-1b3d-406a-aa5b-efcf30e039d4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214319353Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214352639Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214383991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214415373Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214445863Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214477606Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.214509388Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63031)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.215055045Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"ds_connection connected from Ok(127.0.0.1:44152)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.215146886Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"ds_connection connected from Ok(127.0.0.1:41558)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.215182036Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"io task","client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.2168211Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.217006406Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.217610879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.217822166Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.218173238Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[0]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219078515Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[0]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219176067Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219215796Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[1]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219251226Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[1]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219290102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21933473Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[2]R flush_numbers[0..12]: [324, 324, 315, 318, 318, 318, 323, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219384849Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[2]R generation[0..12]: [183, 183, 177, 180, 180, 180, 182, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219434507Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219483294Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219513684Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219556298Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Next flush: 329","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219585776Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219615024Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"":"downstairs","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219653409Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219683919Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"0","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219726303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"1","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:07:51.219758467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"client":"2","":"downstairs","session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21980596Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21983644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"58700fc4-1b3d-406a-aa5b-efcf30e039d4 is now active with session: c687f4ff-be90-458c-97e4-4ab20bf937a5","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219869977Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219897902Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939,"session_id":"c687f4ff-be90-458c-97e4-4ab20bf937a5"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.21992723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} {"msg":"Activated sub_volume 58700fc4-1b3d-406a-aa5b-efcf30e039d4","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:51.219956808Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 334 len 16384 data: 13 13 14 12 02/30 Read block 15 len 36864 03/30 Flush 04/30 Write block 322 len 4096 data: 15 05/30 Write block 242 len 12288 data: 20 20 17 06/30 Read block 344 len 4096 07/30 Read block 258 len 4096 08/30 Read block 546 len 36864 09/30 Write block 399 len 20480 data: 18 20 22 21 25 10/30 Read block 474 len 40960 11/30 Read block 444 len 4096 12/30 Flush 13/30 Write block 311 len 40960 data: 16 16 16 14 18 14 13 14 12 16 14/30 Read block 263 len 24576 15/30 Write block 103 len 24576 data: 13 13 14 12 10 11 16/30 Write block 403 len 4096 data: 26 17/30 Read block 319 len 16384 18/30 Write block 457 len 32768 data: 15 15 10 13 13 13 17 15 19/30 Read block 59 len 20480 20/30 Write block 562 len 12288 data: 11 12 12 21/30 Read block 298 len 4096 22/30 Read block 178 len 40960 23/30 Write block 404 len 36864 data: 21 22 18 11 12 12 14 16 17 24/30 Flush 25/30 Write block 73 len 4096 data: 14 26/30 Flush 27/30 Write block 573 len 16384 data: 11 12 12 14 28/30 Write block 91 len 36864 data: 15 16 14 13 11 8 7 7 8 29/30 Write block 128 len 16384 data: 14 12 16 13 30/30 Read block 250 len 12288 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 9 Done Sent Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 3 28 31 Done 30 27 2 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:51.249194965Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1939} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 324 324 324 F F F 1 020-039 183 183 183 324 324 324 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 186 186 180 332 332 318 F F F <--- 4 080-099 180 180 180 318 318 318 T T F <--- 5 100-119 186 186 180 331 331 318 F F F <--- 6 120-139 182 182 182 323 323 323 T T F <--- 7 140-159 185 185 185 328 328 328 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 184 184 184 327 327 327 F F F 12 240-259 186 186 183 330 330 324 F F F <--- 13 260-279 185 185 185 328 328 328 F F F 14 280-299 185 185 185 328 328 328 F F F 15 300-319 186 186 175 331 331 310 F F F <--- 16 320-339 186 186 183 331 331 324 F F T <--- 17 340-359 182 182 182 321 321 321 F F F 18 360-379 182 182 182 321 321 321 F F F 19 380-399 186 186 182 330 330 321 F F F <--- 20 400-419 186 186 180 331 331 318 F F F <--- 21 420-439 174 174 174 308 308 308 F F F 22 440-459 186 186 184 331 331 327 F F F <--- 23 460-479 186 186 184 331 331 327 F F F <--- 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 185 185 185 328 328 328 F F F 27 540-559 184 184 184 327 327 327 F F F 28 560-579 186 186 184 331 331 327 T T F <--- 29 580-599 184 184 184 327 327 327 F F F Max gen: 186, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 324 324 324 F F F 1 020-039 183 183 183 324 324 324 F F F 2 040-059 177 177 177 315 315 315 F F F 3 060-079 186 186 186 332 332 332 F F F 4 080-099 187 187 187 333 333 333 F F F 5 100-119 186 186 186 331 331 331 F F F 6 120-139 187 187 187 333 333 333 F F F 7 140-159 185 185 185 328 328 328 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 184 184 184 327 327 327 F F F 12 240-259 186 186 186 330 330 330 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 185 185 185 328 328 328 F F F 15 300-319 186 186 186 331 331 331 F F F 16 320-339 186 186 186 331 331 331 F F F 17 340-359 182 182 182 321 321 321 F F F 18 360-379 182 182 182 321 321 321 F F F 19 380-399 186 186 186 330 330 330 F F F 20 400-419 186 186 186 331 331 331 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 186 186 186 331 331 331 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 F F F 26 520-539 185 185 185 328 328 328 F F F 27 540-559 184 184 184 327 327 327 F F F 28 560-579 187 187 187 333 333 333 F F F 29 580-599 184 184 184 327 327 327 F F F Max gen: 187, Max flush: 333 Begin loop 94 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.812389774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.837518242Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.838292961Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.838371015Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.838398279Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.838438618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"Crucible a23637e9-c559-4df2-afae-49d6f4146e22 has session id: fc0082e9-5e2d-4db9-a8f6-0a694df6cd26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.839465384Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Upstairs opts: Upstairs UUID: a23637e9-c559-4df2-afae-49d6f4146e22, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.839531024Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Crucible stats registered with UUID: a23637e9-c559-4df2-afae-49d6f4146e22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.839564159Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:55.839593197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.839997765Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840052303Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840084496Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840590495Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.8406285Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"a23637e9-c559-4df2-afae-49d6f4146e22 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840725441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840757173Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840787884Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840818615Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840848824Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840880266Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.840911538Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"ds_connection connected from Ok(127.0.0.1:38808)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.841291758Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"ds_connection connected from Ok(127.0.0.1:61969)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.841335695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"ds_connection connected from Ok(127.0.0.1:50815)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.841368941Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"io task","client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.843104967Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.843155196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.843307416Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.843938952Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.843982047Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[0]R flush_numbers[0..12]: [324, 324, 315, 332, 333, 331, 333, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.84413025Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[0]R generation[0..12]: [183, 183, 177, 186, 187, 186, 187, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844172082Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844208885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[1]R flush_numbers[0..12]: [324, 324, 315, 332, 333, 331, 333, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844243644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[1]R generation[0..12]: [183, 183, 177, 186, 187, 186, 187, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844277921Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844313942Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[2]R flush_numbers[0..12]: [324, 324, 315, 332, 333, 331, 333, 328, 319, 319, 316, 327]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.84435347Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[2]R generation[0..12]: [183, 183, 177, 186, 187, 186, 187, 185, 181, 181, 178, 184]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844387988Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844423749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844453949Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844484659Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844513636Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844542583ZWait for a query_work_queue command to finish before sending IO ","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"":"downstairs","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844585879Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844613604Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"0","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844644264Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"1","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844674764Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"client":"2","":"downstairs","session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844705725Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844734883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"a23637e9-c559-4df2-afae-49d6f4146e22 is now active with session: fc0082e9-5e2d-4db9-a8f6-0a694df6cd26","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844767618Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844795212Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947,"session_id":"fc0082e9-5e2d-4db9-a8f6-0a694df6cd26"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844827856Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} {"msg":"Activated sub_volume a23637e9-c559-4df2-afae-49d6f4146e22","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:07:55.844853176Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 236 len 8192 data: 13 14 02/30 Read block 234 len 12288 03/30 Read block 582 len 32768 04/30 Read block 187 len 24576 05/30 Write block 415 len 12288 data: 13 14 14 06/30 Write block 561 len 20480 data: 10 12 13 13 15 07/30 Write block 48 len 28672 data: 13 11 13 8 6 8 11 08/30 Read block 423 len 8192 09/30 Write block 359 len 20480 data: 14 14 14 14 16 10/30 Write block 307 len 16384 data: 12 14 13 13 11/30 Read block 415 len 32768 12/30 Write block 285 len 20480 data: 19 18 13 16 18 13/30 Write block 5 len 24576 data: 11 13 12 16 18 21 14/30 Read block 253 len 40960 15/30 Read block 269 len 40960 16/30 Flush 17/30 Write block 556 len 24576 data: 14 17 18 19 12 11 18/30 Write block 506 len 36864 data: 11 14 16 15 17 15 9 9 9 19/30 Write block 43 len 20480 data: 11 11 16 17 12 20/30 Write block 359 len 32768 data: 15 15 15 15 17 19 16 19 21/30 Write block 20 len 36864 data: 14 16 13 11 11 11 11 14 17 22/30 Read block 574 len 20480 23/30 Write block 247 len 16384 data: 16 16 12 10 24/30 Read block 469 len 36864 25/30 Read block 8 len 36864 26/30 Read block 385 len 20480 27/30 Write block 376 len 16384 data: 13 19 22 25 28/30 Read block 549 len 36864 29/30 Write block 403 len 40960 data: 27 22 23 19 12 13 13 15 17 18 30/30 Write block 108 len 16384 data: 12 13 16 15 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Write 7 Done Done Done false 1007 Acked Read 2 Done Done Done false 1008 Acked Write 5 Done Done Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 21 2 25 Done 28 9 28 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:07:55.875178028Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1947} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 183 188 334 324 334 F F F <--- 1 020-039 183 183 183 324 324 324 T F T <--- 2 040-059 188 177 188 334 315 334 T T T <--- 3 060-079 186 186 186 332 332 332 F F F 4 080-099 187 187 187 333 333 333 F F F 5 100-119 186 186 186 331 331 331 T F T <--- 6 120-139 187 187 187 333 333 333 F F F 7 140-159 185 185 185 328 328 328 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 188 184 188 334 327 334 F T F <--- 12 240-259 186 186 186 330 330 330 T F T <--- 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 185 188 334 328 334 F F F <--- 15 300-319 188 186 188 334 331 334 F F F <--- 16 320-339 186 186 186 331 331 331 F F F 17 340-359 188 182 188 334 321 334 T T T <--- 18 360-379 188 182 188 334 321 334 T T T <--- 19 380-399 186 186 186 330 330 330 F F F 20 400-419 188 186 188 334 331 334 T T T <--- 21 420-439 174 174 174 308 308 308 F F F 22 440-459 186 186 186 331 331 331 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 176 176 176 314 314 314 T F T <--- 26 520-539 185 185 185 328 328 328 F F F 27 540-559 184 184 184 327 327 327 T F T <--- 28 560-579 188 187 188 334 333 334 T T T <--- 29 580-599 184 184 184 327 327 327 F F F Max gen: 188, Max flush: 334 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 334 334 334 F F F 1 020-039 189 189 189 335 335 335 F F F 2 040-059 189 189 189 335 335 335 F F F 3 060-079 186 186 186 332 332 332 F F F 4 080-099 187 187 187 333 333 333 F F F 5 100-119 189 189 189 335 335 335 F F F 6 120-139 187 187 187 333 333 333 F F F 7 140-159 185 185 185 328 328 328 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 188 188 188 334 334 334 F F F 12 240-259 189 189 189 335 335 335 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 188 188 188 334 334 334 F F F 16 320-339 186 186 186 331 331 331 F F F 17 340-359 189 189 189 335 335 335 F F F 18 360-379 189 189 189 335 335 335 F F F 19 380-399 186 186 186 330 330 330 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 174 174 174 308 308 308 F F F 22 440-459 186 186 186 331 331 331 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 189 189 189 335 335 335 F F F 26 520-539 185 185 185 328 328 328 F F F 27 540-559 189 189 189 335 335 335 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 184 184 184 327 327 327 F F F Max gen: 189, Max flush: 335 Begin loop 95 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.461158681Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:00.482365319Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.48244662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.499993716Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.50045177Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.500537018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.500564102Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.500604582Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"Crucible 75be817c-5aa8-4c4d-9503-550f23b527db has session id: 7d4e0db6-447f-4b6d-9e32-874c08afe530","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.501831793Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Upstairs opts: Upstairs UUID: 75be817c-5aa8-4c4d-9503-550f23b527db, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.501925649Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Crucible stats registered with UUID: 75be817c-5aa8-4c4d-9503-550f23b527db","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.501960417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:00.501989294Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502454302Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502540622Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502575571Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502793111Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502827148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"75be817c-5aa8-4c4d-9503-550f23b527db active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502941984Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.502974799Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503006381Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503038004Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503068474Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503100106Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503480427Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"ds_connection connected from Ok(127.0.0.1:63728)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503562178Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"ds_connection connected from Ok(127.0.0.1:56353)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503596887Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"ds_connection connected from Ok(127.0.0.1:41629)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.503630333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"io task","client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.504981419Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.50503268Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.505545452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506176888Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506209723Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[0]R flush_numbers[0..12]: [334, 335, 335, 332, 333, 335, 333, 328, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506487882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[0]R generation[0..12]: [188, 189, 189, 186, 187, 189, 187, 185, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.50652763Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506564282Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[1]R flush_numbers[0..12]: [334, 335, 335, 332, 333, 335, 333, 328, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506599452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[1]R generation[0..12]: [188, 189, 189, 186, 187, 189, 187, 185, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506635062Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506671614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[2]R flush_numbers[0..12]: [334, 335, 335, 332, 333, 335, 333, 328, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506706493Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[2]R generation[0..12]: [188, 189, 189, 186, 187, 189, 187, 185, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506741392Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506777804Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506807673Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506838674Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Next flush: 336","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.50687229Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506901919Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"":"downstairs","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506933Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506961356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"0","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.506992197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"1","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507022907Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"client":"2","":"downstairs","session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.5070546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507084148Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"75be817c-5aa8-4c4d-9503-550f23b527db is now active with session: 7d4e0db6-447f-4b6d-9e32-874c08afe530","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507117324Z","Wait for a query_work_queue command to finish before sending IO hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507159337Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955,"session_id":"7d4e0db6-447f-4b6d-9e32-874c08afe530"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507188695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} {"msg":"Activated sub_volume 75be817c-5aa8-4c4d-9503-550f23b527db","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:00.507214145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 436 len 32768 data: 15 17 17 13 13 18 16 18 02/30 Write block 17 len 24576 data: 17 17 14 15 17 14 03/30 Write block 313 len 32768 data: 17 15 19 15 14 15 13 17 04/30 Read block 85 len 36864 05/30 Read block 523 len 4096 06/30 Write block 54 len 36864 data: 12 12 11 11 11 11 11 13 13 07/30 Read block 156 len 24576 08/30 Read block 65 len 8192 09/30 Read block 336 len 36864 10/30 Read block 209 len 32768 11/30 Flush 12/30 Write block 137 len 28672 data: 11 9 8 10 10 10 10 13/30 Read block 496 len 12288 14/30 Write block 594 len 24576 data: 7 7 7 7 5 3 15/30 Read block 195 len 28672 16/30 Read block 384 len 28672 17/30 Write block 35 len 16384 data: 18 13 18 16 18/30 Read block 241 len 12288 19/30 Write block 519 len 24576 data: 14 16 17 17 16 18 20/30 Read block 418 len 24576 21/30 Read block 321 len 32768 22/30 Flush 23/30 Write block 373 len 24576 data: 12 14 14 14 20 23 24/30 Read block 50 len 24576 25/30 Read block 291 len 4096 26/30 Flush 27/30 Write block 339 len 16384 data: 12 11 15 15 28/30 Read block 468 len 20480 29/30 Write block 250 len 12288 data: 11 9 9 30/30 Read block 196 len 28672 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 3 Sent Done Done false 1029 Acked Read 7 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:00.541193719Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1955} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 190 190 334 336 336 T F F <--- 1 020-039 189 190 190 335 337 337 T F F <--- 2 040-059 189 190 190 335 336 336 F F F <--- 3 060-079 186 190 190 332 336 336 F F F <--- 4 080-099 187 187 187 333 333 333 F F F 5 100-119 189 189 189 335 335 335 F F F 6 120-139 187 190 190 333 337 337 F F F <--- 7 140-159 185 190 190 328 337 337 F F F <--- 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 188 188 188 334 334 334 F F F 12 240-259 189 189 189 335 335 335 F T T <--- 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 188 190 190 334 336 336 T F F <--- 16 320-339 186 190 190 331 336 336 T T T <--- 17 340-359 189 189 189 335 335 335 F T T <--- 18 360-379 189 190 190 335 338 338 F F F <--- 19 380-399 186 186 186 330 330 330 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 174 190 190 308 336 336 T F F <--- 22 440-459 186 190 190 331 336 336 T F F <--- 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 189 190 190 335 337 337 F F F <--- 26 520-539 185 190 190 328 337 337 F F F <--- 27 540-559 189 189 189 335 335 335 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 184 190 190 327 337 337 F F F <--- Max gen: 190, Max flush: 338 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 190 336 336 336 F F F 1 020-039 190 190 190 337 337 337 F F F 2 040-059 190 190 190 336 336 336 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 187 187 187 333 333 333 F F F 5 100-119 189 189 189 335 335 335 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 190 190 190 337 337 337 F F F 8 160-179 181 181 181 319 319 319 F F F 9 180-199 181 181 181 319 319 319 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 188 188 188 334 334 334 F F F 12 240-259 191 191 191 339 339 339 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 190 190 190 336 336 336 F F F 16 320-339 191 191 191 339 339 339 F F F 17 340-359 191 191 191 339 339 339 F F F 18 360-379 190 190 190 338 338 338 F F F 19 380-399 186 186 186 330 330 330 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 190 190 190 336 336 336 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 190 190 190 337 337 337 F F F 26 520-539 190 190 190 337 337 337 F F F 27 540-559 189 189 189 335 335 335 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 190 190 190 337 337 337 F F F Max gen: 191, Max flush: 339 Begin loop 96 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.121347683Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.143674993Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.14448426Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.144784032Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.144846164Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.144901644Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"Crucible 5b5df76a-52c4-4d59-8441-e278b58d59cb has session id: cd7fe6a0-e368-4842-8a58-70fc60cf611a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.145851779Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Upstairs opts: Upstairs UUID: 5b5df76a-52c4-4d59-8441-e278b58d59cb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.145986244Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Crucible stats registered with UUID: 5b5df76a-52c4-4d59-8441-e278b58d59cb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146023999Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:05.146049369Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146404299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146468526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146498445Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146832534Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146865038Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"5b5df76a-52c4-4d59-8441-e278b58d59cb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146898875Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146923393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.146954665Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.14698255Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147013842Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147041246Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147228175Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"ds_connection connected from Ok(127.0.0.1:53002)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147329304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45054)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147372239Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"ds_connection connected from Ok(127.0.0.1:55907)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.147448139Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"io task","client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.149518434Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.149605466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.149782315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.150430594Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.150480463Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[0]R flush_numbers[0..12]: [336, 337, 336, 336, 333, 335, 337, 337, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151053955Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[0]R generation[0..12]: [190, 190, 190, 190, 187, 189, 190, 190, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151119063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151159142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[1]R flush_numbers[0..12]: [336, 337, 336, 336, 333, 335, 337, 337, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151195063Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[1]R generation[0..12]: [190, 190, 190, 190, 187, 189, 190, 190, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151237296Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.15127478Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[2]R flush_numbers[0..12]: [336, 337, 336, 336, 333, 335, 337, 337, 319, 319, 316, 334]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151309389Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[2]R generation[0..12]: [190, 190, 190, 190, 187, 189, 190, 190, 181, 181, 178, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151344057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151379777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151409717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151440948Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Next flush: 340","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151481638Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151511998Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"":"downstairs","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.15154333Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151572157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"0","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151603118Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"1","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151633669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"client":"2","":"downstairs","session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.15166484Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:08:05.151694138Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"5b5df76a-52c4-4d59-8441-e278b58d59cb is now active with session: cd7fe6a0-e368-4842-8a58-70fc60cf611a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151739467Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151767943Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963,"session_id":"cd7fe6a0-e368-4842-8a58-70fc60cf611a"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151798243Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} {"msg":"Activated sub_volume 5b5df76a-52c4-4d59-8441-e278b58d59cb","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:05.151824124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 228 len 16384 data: 14 12 12 13 02/30 Flush 03/30 Write block 142 len 28672 data: 11 11 13 11 14 12 13 04/30 Read block 395 len 24576 05/30 Read block 325 len 4096 06/30 Write block 224 len 12288 data: 14 15 14 07/30 Write block 347 len 36864 data: 14 13 11 12 13 13 15 11 13 08/30 Read block 560 len 32768 09/30 Read block 148 len 32768 10/30 Write block 451 len 36864 data: 16 17 19 16 16 13 16 16 11 11/30 Flush 12/30 Write block 382 len 16384 data: 23 22 20 18 13/30 Flush 14/30 Flush 15/30 Write block 558 len 16384 data: 19 20 13 12 16/30 Write block 113 len 28672 data: 18 15 15 17 14 15 13 17/30 Write block 99 len 4096 data: 9 18/30 Read block 554 len 28672 19/30 Read block 93 len 16384 20/30 Read block 158 len 28672 21/30 Read block 63 len 28672 22/30 Flush 23/30 Write block 178 len 20480 data: 15 14 16 17 16 24/30 Write block 593 len 4096 data: 8 25/30 Read block 302 len 40960 26/30 Read block 161 len 36864 27/30 Write block 563 len 32768 data: 14 14 16 15 17 16 15 16 28/30 Read block 316 len 4096 29/30 Write block 140 len 16384 data: 11 11 12 12 30/30 Write block 514 len 40960 data: 10 9 7 8 9 15 17 18 18 17 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 7 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:05.180383142Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1963} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 190 336 336 336 F F F 1 020-039 190 190 190 337 337 337 F F F 2 040-059 190 190 190 336 336 336 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 192 187 192 344 333 344 F F F <--- 5 100-119 192 189 192 344 335 344 F F F <--- 6 120-139 190 190 190 337 337 337 F F F 7 140-159 192 190 192 341 337 341 T F T <--- 8 160-179 181 181 181 319 319 319 T F T <--- 9 180-199 181 181 181 319 319 319 T F T <--- 10 200-219 178 178 178 316 316 316 F F F 11 220-239 192 188 192 341 334 341 F F F <--- 12 240-259 191 191 191 339 339 339 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 190 190 190 336 336 336 F F F 16 320-339 191 191 191 339 339 339 F F F 17 340-359 192 191 192 341 339 341 F F F <--- 18 360-379 190 190 190 338 338 338 F F F 19 380-399 192 186 192 342 330 342 F F F <--- 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 192 190 192 341 336 341 F F F <--- 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 190 190 190 337 337 337 T F T <--- 26 520-539 190 190 190 337 337 337 T F T <--- 27 540-559 192 189 192 344 335 344 F F F <--- 28 560-579 192 189 192 344 335 344 T F T <--- 29 580-599 190 190 190 337 337 337 T F T <--- Max gen: 192, Max flush: 344 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 190 336 336 336 F F F 1 020-039 190 190 190 337 337 337 F F F 2 040-059 190 190 190 336 336 336 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 192 192 192 344 344 344 F F F 5 100-119 192 192 192 344 344 344 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 193 193 193 345 345 345 F F F 8 160-179 193 193 193 345 345 345 F F F 9 180-199 193 193 193 345 345 345 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 191 191 191 339 339 339 F F F 13 260-279 185 185 185 328 328 328 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 190 190 190 336 336 336 F F F 16 320-339 191 191 191 339 339 339 F F F 17 340-359 192 192 192 341 341 341 F F F 18 360-379 190 190 190 338 338 338 F F F 19 380-399 192 192 192 342 342 342 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 192 192 192 341 341 341 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 193 193 193 345 345 345 F F F 26 520-539 193 193 193 345 345 345 F F F 27 540-559 192 192 192 344 344 344 F F F 28 560-579 193 193 193 345 345 345 F F F 29 580-599 193 193 193 345 345 345 F F F Max gen: 193, Max flush: 345 Begin loop 97 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.76596157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:09.784406418Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.784485775Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.801220258Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.801661158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.801740645Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.801764382Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.801798749Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"Crucible c0beedc8-5fed-4539-b255-397406448b40 has session id: 0bbf28d8-3fb5-4c9e-abd4-86af032f897a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.802812089Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Upstairs opts: Upstairs UUID: c0beedc8-5fed-4539-b255-397406448b40, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.802937927Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Crucible stats registered with UUID: c0beedc8-5fed-4539-b255-397406448b40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.802972305Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:09.80303018Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803412223Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803470699Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803500828Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803752866Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.80380043Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"c0beedc8-5fed-4539-b255-397406448b40 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803931508Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.803961938Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804015404Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804045774Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804073358Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804102365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804131823Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"ds_connection connected from Ok(127.0.0.1:51613)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804481093Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41678)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804558145Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"ds_connection connected from Ok(127.0.0.1:55457)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.804590539Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"io task","client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.805951945Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.806200826Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.806333669Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.806753818Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.806884315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[0]R flush_numbers[0..12]: [336, 337, 336, 336, 344, 344, 337, 345, 345, 345, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807348301Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[0]R generation[0..12]: [190, 190, 190, 190, 192, 192, 190, 193, 193, 193, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807446695Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807494379Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[1]R flush_numbers[0..12]: [336, 337, 336, 336, 344, 344, 337, 345, 345, 345, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807534638Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[1]R generation[0..12]: [190, 190, 190, 190, 192, 192, 190, 193, 193, 193, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807574918Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807615097Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[2]R flush_numbers[0..12]: [336, 337, 336, 336, 344, 344, 337, 345, 345, 345, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807652661Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[2]R generation[0..12]: [190, 190, 190, 190, 192, 192, 190, 193, 193, 193, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807694304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807736417Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807779452Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Generation requested: 194 >= found:194",Wait for a query_work_queue command to finish before sending IO "v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807827807Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Next flush: 346","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807863848Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807895911Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"":"downstairs","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807932123Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.807964256Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"0","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808000157Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"1","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808035607Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"client":"2","":"downstairs","session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808071197Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808110345Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"c0beedc8-5fed-4539-b255-397406448b40 is now active with session: 0bbf28d8-3fb5-4c9e-abd4-86af032f897a","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808161526Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808190192Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971,"session_id":"0bbf28d8-3fb5-4c9e-abd4-86af032f897a"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808217125Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} {"msg":"Activated sub_volume c0beedc8-5fed-4539-b255-397406448b40","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:09.808241614Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 187 len 40960 data: 16 16 14 15 14 12 10 15 16 16 02/30 Read block 155 len 40960 03/30 Write block 335 len 36864 data: 14 15 13 12 13 12 16 16 13 04/30 Read block 6 len 24576 05/30 Read block 411 len 36864 06/30 Read block 538 len 28672 07/30 Read block 565 len 28672 08/30 Write block 248 len 16384 data: 17 13 12 10 09/30 Read block 11 len 28672 10/30 Read block 457 len 32768 11/30 Write block 581 len 40960 data: 10 11 13 14 13 12 12 11 10 9 12/30 Write block 53 len 12288 data: 9 13 13 13/30 Read block 33 len 28672 14/30 Read block 288 len 28672 15/30 Read block 99 len 36864 16/30 Write block 450 len 24576 data: 15 17 18 20 17 17 17/30 Flush 18/30 Write block 9 len 8192 data: 19 22 19/30 Write block 322 len 12288 data: 16 14 14 20/30 Read block 408 len 12288 21/30 Write block 577 len 28672 data: 12 11 15 9 11 12 14 22/30 Read block 117 len 16384 23/30 Write block 257 len 16384 data: 14 16 18 16 24/30 Write block 104 len 12288 data: 14 15 13 25/30 Read block 123 len 20480 26/30 Write block 456 len 16384 data: 14 17 17 12 27/30 Read block 522 len 32768 28/30 Write block 376 len 12288 data: 15 21 24 29/30 Read block 246 len 16384 30/30 Read block 223 len 20480 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Read 10 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:09.840096385Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1971} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 190 336 336 336 F T T <--- 1 020-039 190 190 190 337 337 337 F F F 2 040-059 190 194 194 336 346 346 F F F <--- 3 060-079 190 190 190 336 336 336 F F F 4 080-099 192 192 192 344 344 344 F F F 5 100-119 192 192 192 344 344 344 F T T <--- 6 120-139 190 190 190 337 337 337 F F F 7 140-159 193 193 193 345 345 345 F F F 8 160-179 193 193 193 345 345 345 F F F 9 180-199 193 194 194 345 346 346 F F F <--- 10 200-219 178 178 178 316 316 316 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 191 194 194 339 346 346 F T T <--- 13 260-279 185 185 185 328 328 328 F T T <--- 14 280-299 188 188 188 334 334 334 F F F 15 300-319 190 190 190 336 336 336 F F F 16 320-339 191 194 194 339 346 346 F T T <--- 17 340-359 192 194 194 341 346 346 F F F <--- 18 360-379 190 190 190 338 338 338 F T T <--- 19 380-399 192 192 192 342 342 342 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 192 194 194 341 346 346 F T T <--- 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 193 193 193 345 345 345 F F F 26 520-539 193 193 193 345 345 345 F F F 27 540-559 192 192 192 344 344 344 F F F 28 560-579 193 193 193 345 345 345 F T T <--- 29 580-599 193 194 194 345 346 346 F T T <--- Max gen: 194, Max flush: 346 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 195 195 195 347 347 347 F F F 1 020-039 190 190 190 337 337 337 F F F 2 040-059 194 194 194 346 346 346 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 192 192 192 344 344 344 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 193 193 193 345 345 345 F F F 8 160-179 193 193 193 345 345 345 F F F 9 180-199 194 194 194 346 346 346 F F F 10 200-219 178 178 178 316 316 316 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 195 195 195 347 347 347 F F F 13 260-279 195 195 195 347 347 347 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 190 190 190 336 336 336 F F F 16 320-339 195 195 195 347 347 347 F F F 17 340-359 194 194 194 346 346 346 F F F 18 360-379 195 195 195 347 347 347 F F F 19 380-399 192 192 192 342 342 342 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 195 195 195 347 347 347 F F F 23 460-479 186 186 186 331 331 331 F F F 24 480-499 183 183 183 324 324 324 F F F 25 500-519 193 193 193 345 345 345 F F F 26 520-539 193 193 193 345 345 345 F F F 27 540-559 192 192 192 344 344 344 F F F 28 560-579 195 195 195 347 347 347 F F F 29 580-599 195 195 195 347 347 347 F F F Max gen: 195, Max flush: 347 Begin loop 98 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.422158806Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.442135547Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.442665713Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.442752234Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.442777183Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.442819086Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"Crucible afbb49a4-9398-4a38-9d79-f29c1d8765aa has session id: 8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.443965308Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Upstairs opts: Upstairs UUID: afbb49a4-9398-4a38-9d79-f29c1d8765aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444027601Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Crucible stats registered with UUID: afbb49a4-9398-4a38-9d79-f29c1d8765aa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444060135Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:14.444086717Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444464323Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444587315Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444638777Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444821477Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.444869222Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"afbb49a4-9398-4a38-9d79-f29c1d8765aa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.44509662Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445148232Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445199704Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445247057Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445291435Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445334289Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445378767Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"ds_connection connected from Ok(127.0.0.1:62310)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445782293Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43914)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445841099Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"ds_connection connected from Ok(127.0.0.1:60366)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.445891058Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"io task","client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.447798883Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.447905934Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.447951254Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448531338Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448602378Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[0]R flush_numbers[0..12]: [347, 337, 346, 336, 344, 347, 337, 345, 345, 346, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448835458Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[0]R generation[0..12]: [195, 190, 194, 190, 192, 195, 190, 193, 193, 194, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448875598Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448916068Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[1]R flush_numbers[0..12]: [347, 337, 346, 336, 344, 347, 337, 345, 345, 346, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448948522Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[1]R generation[0..12]: [195, 190, 194, 190, 192, 195, 190, 193, 193, 194, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.448980204Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.44901336Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[2]R flush_numbers[0..12]: [347, 337, 346, 336, 344, 347, 337, 345, 345, 346, 316, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449045563Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[2]R generation[0..12]: [195, 190, 194, 190, 192, 195, 190, 193, 193, 194, 178, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449081885Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449116072Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449143196Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449171191Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Next flush: 348","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449197643Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449224105Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"":"downstairs","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449252161Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-06T06:08:14.449277882Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"0","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449312861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"1","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449341136Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"client":"2","":"downstairs","session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449376115Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449403319Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"afbb49a4-9398-4a38-9d79-f29c1d8765aa is now active with session: 8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449433098Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.449458277Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979,"session_id":"8dd92dc4-78ea-4a5a-a361-2bc7b989ef0b"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-06T06:08:14.44948491Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} {"msg":"Activated sub_volume afbb49a4-9398-4a38-9d79-f29c1d8765aa","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:14.44951546Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 213 len 8192 data: 11 11 02/30 Write block 89 len 16384 data: 14 16 16 17 03/30 Read block 164 len 36864 04/30 Read block 515 len 40960 05/30 Write block 277 len 4096 data: 25 06/30 Read block 484 len 36864 07/30 Write block 499 len 40960 data: 14 14 14 14 13 14 11 12 15 17 08/30 Write block 23 len 12288 data: 12 12 12 09/30 Write block 442 len 12288 data: 17 19 15 10/30 Read block 277 len 28672 11/30 Read block 382 len 36864 12/30 Write block 514 len 28672 data: 11 10 8 9 10 16 18 13/30 Write block 210 len 8192 data: 8 7 14/30 Write block 376 len 28672 data: 16 22 25 26 24 27 24 15/30 Write block 519 len 16384 data: 17 19 19 19 16/30 Write block 251 len 40960 data: 11 10 11 11 10 13 15 17 19 17 17/30 Read block 332 len 16384 18/30 Write block 168 len 32768 data: 13 14 16 17 14 12 10 10 19/30 Write block 313 len 4096 data: 18 20/30 Write block 18 len 28672 data: 18 15 16 18 15 13 13 21/30 Flush 22/30 Write block 446 len 12288 data: 18 17 18 23/30 Write block 408 len 32768 data: 14 14 16 18 19 13 17 14 24/30 Write block 568 len 8192 data: 17 16 25/30 Read block 413 len 36864 26/30 Read block 558 len 12288 27/30 Write block 511 len 32768 data: 16 10 10 12 11 9 10 11 28/30 Write block 468 len 40960 data: 14 13 15 18 16 17 17 17 15 15 29/30 Write block 561 len 8192 data: 13 13 30/30 Read block 446 len 20480 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Read 9 Done Done Done false 1006 Acked Write 10 Done Done Done false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 9 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 23 24 Done 29 30 7 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:14.473516393Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1979} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 196 196 195 348 348 347 F F F <--- 1 020-039 196 196 190 348 348 337 F F F <--- 2 040-059 194 194 194 346 346 346 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 196 196 192 348 348 344 F F T <--- 5 100-119 195 195 195 347 347 347 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 193 193 193 345 345 345 F F F 8 160-179 196 196 193 348 348 345 F F F <--- 9 180-199 194 194 194 346 346 346 F F F 10 200-219 196 196 178 348 348 316 F F T <--- 11 220-239 192 192 192 341 341 341 F F F 12 240-259 196 196 195 348 348 347 F F F <--- 13 260-279 196 196 195 348 348 347 F F T <--- 14 280-299 188 188 188 334 334 334 F F F 15 300-319 196 196 190 348 348 336 F F F <--- 16 320-339 195 195 195 347 347 347 F F F 17 340-359 194 194 194 346 346 346 F F F 18 360-379 196 196 195 348 348 347 F F F <--- 19 380-399 196 196 192 348 348 342 F F F <--- 20 400-419 189 189 189 335 335 335 T T F <--- 21 420-439 190 190 190 336 336 336 F F F 22 440-459 196 196 195 348 348 347 T T F <--- 23 460-479 186 186 186 331 331 331 T T F <--- 24 480-499 196 196 183 348 348 324 F F T <--- 25 500-519 196 196 193 348 348 345 T T T <--- 26 520-539 196 196 193 348 348 345 F F F <--- 27 540-559 192 192 192 344 344 344 F F F 28 560-579 195 195 195 347 347 347 T T F <--- 29 580-599 195 195 195 347 347 347 F F F Max gen: 196, Max flush: 348 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 196 196 196 348 348 348 F F F 1 020-039 196 196 196 348 348 348 F F F 2 040-059 194 194 194 346 346 346 F F F 3 060-079 190 190 190 336 336 336 F F F 4 080-099 196 196 196 348 348 348 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 193 193 193 345 345 345 F F F 8 160-179 196 196 196 348 348 348 F F F 9 180-199 194 194 194 346 346 346 F F F 10 200-219 196 196 196 348 348 348 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 196 196 196 348 348 348 F F F 13 260-279 196 196 196 348 348 348 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 196 196 196 348 348 348 F F F 16 320-339 195 195 195 347 347 347 F F F 17 340-359 194 194 194 346 346 346 F F F 18 360-379 196 196 196 348 348 348 F F F 19 380-399 196 196 196 348 348 348 F F F 20 400-419 197 197 197 349 349 349 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 197 197 197 349 349 349 F F F 23 460-479 197 197 197 349 349 349 F F F 24 480-499 196 196 196 348 348 348 F F F 25 500-519 197 197 197 349 349 349 F F F 26 520-539 196 196 196 348 348 348 F F F 27 540-559 192 192 192 344 344 344 F F F 28 560-579 197 197 197 349 349 349 F F F 29 580-599 195 195 195 347 347 347 F F F Max gen: 197, Max flush: 349 Begin loop 99 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.031109466Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.051470426Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.051973328Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"c1bb7fdf8831e250ccda9b3d318506e60bbc9a46\",\n git_commit_timestamp: \"2026-03-06T05:33:38.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.94.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4a4ef493e3a1488c6e321570238084b38948f6db\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.052050821Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.052076762Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.052113304Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"Crucible ceca800c-7c2d-4bd8-9a0b-49ba44e31b69 has session id: 6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.05324119Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Upstairs opts: Upstairs UUID: ceca800c-7c2d-4bd8-9a0b-49ba44e31b69, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.053302862Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Crucible stats registered with UUID: ceca800c-7c2d-4bd8-9a0b-49ba44e31b69","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.053338562Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:19.053365405Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.053772228Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.053847997Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.053873468Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054244861Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054278908Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"ceca800c-7c2d-4bd8-9a0b-49ba44e31b69 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054382803Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054408594Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054431299Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054453332Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054475566Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054505365Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054770668Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:47820)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054800447Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:45763)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.054824384Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:50893)","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.055111461Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"io task","client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.056774583Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.056817688Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.056844441Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057604831Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057724247Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[0]R flush_numbers[0..12]: [348, 348, 346, 336, 348, 347, 337, 345, 348, 346, 348, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057814124Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[0]R generation[0..12]: [196, 196, 194, 190, 196, 195, 190, 193, 196, 194, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057844604Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057870996Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[1]R flush_numbers[0..12]: [348, 348, 346, 336, 348, 347, 337, 345, 348, 346, 348, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057896416Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[1]R generation[0..12]: [196, 196, 194, 190, 196, 195, 190, 193, 196, 194, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057921356Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057947457Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[2]R flush_numbers[0..12]: [348, 348, 346, 336, 348, 347, 337, 345, 348, 346, 348, 341]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057972046Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[2]R generation[0..12]: [196, 196, 194, 190, 196, 195, 190, 193, 196, 194, 196, 192]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.057996684Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058027786Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.05805004Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058072364Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Next flush: 350","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058093586Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058114587Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"":"downstairs","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058136761Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058156991Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"0","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058179144Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"1","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058200567Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"client":"2","":"downstairs","session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.05822253Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058243511Z","hostname":"Wait for a query_work_queue command to finish before sending IO bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"ceca800c-7c2d-4bd8-9a0b-49ba44e31b69 is now active with session: 6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.05827343Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.05829349Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987,"session_id":"6e9b02d4-0dc2-45fa-95f3-7981c6dbb7a6"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058314652Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} {"msg":"Activated sub_volume ceca800c-7c2d-4bd8-9a0b-49ba44e31b69","v":0,"name":"crucible","level":30,"time":"2026-03-06T06:08:19.058333158Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 566 len 40960 data: 16 18 18 17 17 13 12 12 13 13 02/30 Read block 572 len 40960 03/30 Flush 04/30 Write block 78 len 36864 data: 20 17 19 22 23 20 17 17 16 05/30 Write block 546 len 24576 data: 13 14 13 18 16 17 06/30 Write block 344 len 8192 data: 11 10 07/30 Write block 509 len 8192 data: 16 18 08/30 Read block 499 len 4096 09/30 Write block 441 len 4096 data: 19 10/30 Read block 178 len 28672 11/30 Read block 215 len 32768 12/30 Write block 162 len 20480 data: 18 16 15 12 13 13/30 Write block 455 len 32768 data: 18 15 18 18 13 14 14 14 14/30 Write block 159 len 16384 data: 10 13 15 19 15/30 Flush 16/30 Flush 17/30 Write block 86 len 36864 data: 17 13 15 15 17 17 18 15 14 18/30 Write block 520 len 20480 data: 20 20 20 18 19 19/30 Read block 355 len 16384 20/30 Read block 53 len 16384 21/30 Read block 361 len 12288 22/30 Write block 76 len 4096 data: 20 23/30 Read block 237 len 12288 24/30 Write block 50 len 16384 data: 14 9 7 10 25/30 Write block 405 len 20480 data: 24 20 13 15 15 26/30 Read block 385 len 20480 27/30 Write block 347 len 4096 data: 15 28/30 Read block 77 len 12288 29/30 Write block 450 len 36864 data: 16 18 19 21 18 19 16 19 19 30/30 Write block 96 len 12288 data: 9 8 8 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 10 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 9 Done Sent Done false 1004 Acked Write 6 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-06T06:08:19.08468003Z","hostname":"bmat-EVT22200007-0000ec3a","pid":1987} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 196 196 196 348 348 348 F F F 1 020-039 196 196 196 348 348 348 F F F 2 040-059 194 194 194 346 346 346 T F T <--- 3 060-079 198 190 198 351 336 351 T F T <--- 4 080-099 198 196 198 351 348 351 T F T <--- 5 100-119 195 195 195 347 347 347 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 198 193 198 351 345 351 F F F <--- 8 160-179 198 196 198 351 348 351 F F F <--- 9 180-199 194 194 194 346 346 346 F F F 10 200-219 196 196 196 348 348 348 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 196 196 196 348 348 348 F F F 13 260-279 196 196 196 348 348 348 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 196 196 196 348 348 348 F F F 16 320-339 195 195 195 347 347 347 F F F 17 340-359 198 194 198 351 346 351 T F T <--- 18 360-379 196 196 196 348 348 348 F F F 19 380-399 196 196 196 348 348 348 F F F 20 400-419 197 197 197 349 349 349 T F T <--- 21 420-439 190 190 190 336 336 336 F F F 22 440-459 198 197 198 351 349 351 T F T <--- 23 460-479 198 197 198 351 349 351 F F F <--- 24 480-499 196 196 196 348 348 348 F F F 25 500-519 198 197 198 351 349 351 F F F <--- 26 520-539 196 196 196 348 348 348 T F T <--- 27 540-559 198 192 198 351 344 351 F F F <--- 28 560-579 198 197 198 350 349 350 F T F <--- 29 580-599 195 195 195 347 347 347 F F F Max gen: 198, Max flush: 351 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 196 196 196 348 348 348 F F F 1 020-039 196 196 196 348 348 348 F F F 2 040-059 199 199 199 352 352 352 F F F 3 060-079 199 199 199 352 352 352 F F F 4 080-099 199 199 199 352 352 352 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 190 190 190 337 337 337 F F F 7 140-159 198 198 198 351 351 351 F F F 8 160-179 198 198 198 351 351 351 F F F 9 180-199 194 194 194 346 346 346 F F F 10 200-219 196 196 196 348 348 348 F F F 11 220-239 192 192 192 341 341 341 F F F 12 240-259 196 196 196 348 348 348 F F F 13 260-279 196 196 196 348 348 348 F F F 14 280-299 188 188 188 334 334 334 F F F 15 300-319 196 196 196 348 348 348 F F F 16 320-339 195 195 195 347 347 347 F F F 17 340-359 199 199 199 352 352 352 F F F 18 360-379 196 196 196 348 348 348 F F F 19 380-399 196 196 196 348 348 348 F F F 20 400-419 199 199 199 352 352 352 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 199 199 199 352 352 352 F F F 23 460-479 198 198 198 351 351 351 F F F 24 480-499 196 196 196 348 348 348 F F F 25 500-519 198 198 198 351 351 351 F F F 26 520-539 199 199 199 352 352 352 F F F 27 540-559 198 198 198 351 351 351 F F F 28 560-579 198 198 198 350 350 350 F F F 29 580-599 195 195 195 347 347 347 F F F Max gen: 199, Max flush: 352 9:44 Test duration Test completed real 9:44.394982074 user 1:06.293371422 sys 37.683131349 trap 0.148477845 tflt 0.287678850 dflt 0.377384109 kflt 0.015212347 lock 15:31:40.112074172 slp 46:56.978058245 lat 15.299689791 stop 0.058075001