Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.20697638 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.014423713 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.012568555 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1134 1135 1136 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.42622429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.442588894Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.445021652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.445127073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.445135982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.445147531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Crucible b70ae37b-5ee6-4041-8bbe-21d000c189bf has session id: da3737c3-9059-4bcc-a168-a349c2a3baf3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.44596506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Upstairs opts: Upstairs UUID: b70ae37b-5ee6-4041-8bbe-21d000c189bf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.446313219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Crucible stats registered with UUID: b70ae37b-5ee6-4041-8bbe-21d000c189bf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.446323628Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:09.446369334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447098811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.44733521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447349679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447717087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447761993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"b70ae37b-5ee6-4041-8bbe-21d000c189bf active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447770482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447779052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447788541Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.44779789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447807329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447817058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447852215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47549)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447924449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:46402)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447953116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"ds_connection connected from Ok(127.0.0.1:51707)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.447965465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.450142096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.450165994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.450182962Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451144618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451154048Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451225941Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.45123668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451247689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451257649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451267618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451278327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451288106Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451298095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451308614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451317843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451327253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451336382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451346731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"downstairs","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.45135619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451364669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451374238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451383818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"client":"2","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451393547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451402536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"b70ae37b-5ee6-4041-8bbe-21d000c189bf is now active with session: da3737c3-9059-4bcc-a168-a349c2a3baf3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451412195Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451420634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451429454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Activated sub_volume b70ae37b-5ee6-4041-8bbe-21d000c189bf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.451909592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.45193276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Fill test","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.452242593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} Read and Verify all blocks (0..600 range:false) {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30---------------------------------------------------------------- ,"time":"2026-09-18T03:28:09.492781201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"CLIENT: Tests done. All submitted work has been ACK'd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.492818598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 256 Done Done Done false 1005 Acked Read 88 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 {"msg":"CLIENT: Up:0 ds:3 act:3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:09.492951286Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 {"msg":"CLIENT: Up:0 ds:0 act:3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.49449014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.494563284Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"task":"crutest"} {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.494583792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139} {"msg":"client stopping due to Err(RecvError(()))","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.49460793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.494654226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.494676104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.494696852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client request queue closed unexpectedly; is the program exiting?","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.49472072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.494741888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.494763246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.494783884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"da3737c3-9059-4bcc-a168-a349c2a3baf3"} Begin loop 1 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.520131646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528323132Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528601768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528625736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528634105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528645394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Crucible 8bed6795-e868-4795-830f-8cdac56892d2 has session id: 266a0e70-ce39-4a44-8da1-0dd66198b066","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.528979175Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Upstairs opts: Upstairs UUID: 8bed6795-e868-4795-830f-8cdac56892d2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529015702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Crucible stats registered with UUID: 8bed6795-e868-4795-830f-8cdac56892d2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529026441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.52903552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529167129Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529196646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529207245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529339214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529362392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"8bed6795-e868-4795-830f-8cdac56892d2 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529371161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.52938002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529392319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529401888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529411218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529421067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529430786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"ds_connection connected from Ok(127.0.0.1:51547)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529490111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"ds_connection connected from Ok(127.0.0.1:34533)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529514899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"ds_connection connected from Ok(127.0.0.1:53300)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.529525888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.530156873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.53018152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.530455927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.530472525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.530774739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531043005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531054214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531067843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531078062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531087912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531098671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.53110862Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531118569Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531129258Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531138727Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531148336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531157375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531166605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"":"downstairs","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531176064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531184783Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"0","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531194152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"1","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531203521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"client":"2","":"downstairs","session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.53121458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.53122353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"8bed6795-e868-4795-830f-8cdac56892d2 is now active with session: 266a0e70-ce39-4a44-8da1-0dd66198b066","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531233369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531241968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"session_id":"266a0e70-ce39-4a44-8da1-0dd66198b066"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531250717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Activated sub_volume 8bed6795-e868-4795-830f-8cdac56892d2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531258617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.531266716Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.53179239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} 01/30 Write block 48 len 32768 data: 2 2 2 2 2 2 2 2 02/30 Write block 253 len 36864 data: 2 2 2 2 2 2 2 2 2 03/30 Write block 261 len 28672 data: 3 2 2 2 2 2 2 04/30 Read block 420 len 40960 05/30 Read block 190 len 12288 06/30 Flush 07/30 Write block 360 len 40960 data: 2 2 2 2 2 2 2 2 2 2 08/30 Write block 258 len 4096 data: 3 09/30 Write block 99 len 28672 data: 2 2 2 2 2 2 2 10/30 Read block 428 len 16384 11/30 Read block 211 len 8192 12/30 Read block 216 len 4096 13/30 Flush 14/30 Write block 183 len 24576 data: 2 2 2 2 2 2 15/30 Read block 166 len 28672 16/30 Write block 24 len 20480 data: 2 2 2 2 2 17/30 Read block 146 len 24576 18/30 Write block 502 len 32768 data: 2 2 2 2 2 2 2 2 19/30 Read block 582 len 40960 20/30 Write block 587 len 24576 data: 2 2 2 2 2 2 21/30 Read block 298 len 36864 22/30 Write block 474 len 32768 data: 2 2 2 2 2 2 2 2 23/30 Write block 416 len 12288 data: 2 2 2 24/30 Read block 98 len 24576 25/30 Flush 26/30 Write block 98 len 12288 data: 2 3 3 27/30 Read block 409 len 40960 28/30 Read block 206 len 28672 29/30 Write block 524 len 16384 data: 2 2 2 2 30/30 Write block 350 len 4096 data: 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Read 10 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 27 31 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:13.540830413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:13.540992969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1141,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 2 2 1 4 4 1 F F F <--- 2 040-059 2 2 1 2 2 1 F F T <--- 3 060-079 1 1 1 1 1 1 F F F 4 080-099 2 2 1 3 3 1 T T F <--- 5 100-119 2 2 1 3 3 1 T T F <--- 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 1 4 4 1 F F F <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 1 3 3 1 F F T <--- 13 260-279 2 2 1 2 2 1 F F T <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 1 1 1 1 1 1 T T F <--- 18 360-379 2 2 1 3 3 1 F F F <--- 19 380-399 1 1 1 1 1 1 F F F 20 400-419 2 2 1 4 4 1 F F F <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 1 4 4 1 F F F <--- 24 480-499 2 2 1 4 4 1 F F F <--- 25 500-519 2 2 1 4 4 1 F F F <--- 26 520-539 1 1 1 1 1 1 T T F <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 2 2 1 4 4 1 F F F <--- Max gen: 2, Max flush: 4 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 2 2 2 4 4 4 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 3 3 3 5 5 5 F F F 5 100-119 3 3 3 5 5 5 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 2 3 3 3 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 3 3 3 5 5 5 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 4 4 4 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 2 2 2 4 4 4 F F F 26 520-539 3 3 3 5 5 5 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 2 2 2 4 4 4 F F F Max gen: 3, Max flush: 5 Begin loop 2 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.91983692Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:17.927066341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.927120376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934161442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934412031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934437048Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934445328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934456787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Crucible fe54209c-0be2-48e3-bfbb-0d89c81a1f47 has session id: 51a9961a-4031-429c-bbb5-953d607f7ac5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.93476234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Upstairs opts: Upstairs UUID: fe54209c-0be2-48e3-bfbb-0d89c81a1f47, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934791477Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Crucible stats registered with UUID: fe54209c-0be2-48e3-bfbb-0d89c81a1f47","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934802597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:17.934811676Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934929285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934946324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.934956323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.93511241Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935127748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"fe54209c-0be2-48e3-bfbb-0d89c81a1f47 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935150626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935159875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935169235Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935178834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935188183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935197772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935207491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:43718)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935284045Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63368)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935308802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:52742)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935319891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935950047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935977114Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.935988163Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.93625687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936293207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[0]R flush_numbers[0..12]: [1, 4, 2, 1, 5, 5, 1, 1, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936362981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[0]R generation[0..12]: [1, 2, 2, 1, 3, 3, 1, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.93637375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936384609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[1]R flush_numbers[0..12]: [1, 4, 2, 1, 5, 5, 1, 1, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936394688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[1]R generation[0..12]: [1, 2, 2, 1, 3, 3, 1, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936404587Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936415156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[2]R flush_numbers[0..12]: [1, 4, 2, 1, 5, 5, 1, 1, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936427345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[2]R generation[0..12]: [1, 2, 2, 1, 3, 3, 1, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936437174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936447883Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936456992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936466492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936475531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.93648472Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"":"downstairs","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936494359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936502938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"0","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936512448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"1","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936521747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"client":"2","":"downstairs","session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936531446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936540315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"fe54209c-0be2-48e3-bfbb-0d89c81a1f47 is now active with session: 51a9961a-4031-429c-bbb5-953d607f7ac5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936549914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936558374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"session_id":"51a9961a-4031-429c-bbb5-953d607f7ac5"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936567343Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Activated sub_volume fe54209c-0be2-48e3-bfbb-0d89c81a1f47","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936576392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936584281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.936902374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} 01/30 Write block 28 len 4096 data: 3 02/30 Read block 409 len 4096 03/30 Read block 280 len 40960 04/30 Write block 316 len 36864 data: 2 2 2 2 2 2 2 2 2 05/30 Read block 378 len 12288 06/30 Read block 439 len 8192 07/30 Write block 54 len 4096 data: 3 08/30 Read block 505 len 16384 09/30 Write block 398 len 40960 data: 2 2 2 2 2 2 2 2 2 2 10/30 Write block 437 len 8192 data: 2 2 11/30 Write block 492 len 20480 data: 2 2 2 2 2 12/30 Write block 439 len 24576 data: 2 2 2 2 2 2 13/30 Read block 498 len 8192 14/30 Flush 15/30 Write block 583 len 36864 data: 2 2 2 2 3 3 3 3 3 16/30 Write block 140 len 24576 data: 2 2 2 2 2 2 17/30 Read block 471 len 4096 18/30 Read block 582 len 40960 19/30 Write block 60 len 36864 data: 2 2 2 2 2 2 2 2 2 20/30 Read block 291 len 12288 21/30 Read block 38 len 8192 22/30 Read block 156 len 8192 23/30 Write block 586 len 28672 data: 3 4 4 4 4 4 3 24/30 Write block 481 len 16384 data: 3 2 2 2 25/30 Read block 239 len 40960 26/30 Flush 27/30 Write block 246 len 32768 data: 2 2 2 2 2 2 2 3 28/30 Write block 128 len 8192 data: 2 2 29/30 Write block 31 len 36864 data: 2 2 2 2 2 2 2 2 2 30/30 Read block 42 len 12288 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 1 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Read 3 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 3 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:17.945070722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:17.945240377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1149,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 2 4 4 4 6 6 T T T <--- 2 040-059 2 4 4 2 6 6 F F F <--- 3 060-079 1 4 4 1 7 7 F F F <--- 4 080-099 3 3 3 5 5 5 F F F 5 100-119 3 3 3 5 5 5 F F F 6 120-139 1 1 1 1 1 1 F T T <--- 7 140-159 1 4 4 1 7 7 F F F <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 2 2 2 3 3 3 F T T <--- 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 4 4 1 6 6 F F F <--- 16 320-339 1 4 4 1 6 6 F F F <--- 17 340-359 3 3 3 5 5 5 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 1 4 4 1 6 6 F F F <--- 20 400-419 2 4 4 4 6 6 F F F <--- 21 420-439 1 4 4 1 6 6 F F F <--- 22 440-459 1 4 4 1 6 6 F F F <--- 23 460-479 2 2 2 4 4 4 F F F 24 480-499 2 4 4 4 7 7 F F F <--- 25 500-519 2 2 2 4 4 4 F F F 26 520-539 3 3 3 5 5 5 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 2 4 4 4 7 7 F F F <--- Max gen: 4, Max flush: 7 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 5 5 5 8 8 8 F F F 2 040-059 4 4 4 6 6 6 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 3 3 3 5 5 5 F F F 5 100-119 3 3 3 5 5 5 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 4 4 4 7 7 7 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 5 5 5 8 8 8 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 4 4 4 6 6 6 F F F 16 320-339 4 4 4 6 6 6 F F F 17 340-359 3 3 3 5 5 5 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 4 4 4 6 6 6 F F F 20 400-419 4 4 4 6 6 6 F F F 21 420-439 4 4 4 6 6 6 F F F 22 440-459 4 4 4 6 6 6 F F F 23 460-479 2 2 2 4 4 4 F F F 24 480-499 4 4 4 7 7 7 F F F 25 500-519 2 2 2 4 4 4 F F F 26 520-539 3 3 3 5 5 5 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 4 4 4 7 7 7 F F F Max gen: 5, Max flush: 8 Begin loop 3 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.197148498Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.205466484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.205748319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.205778646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.205787256Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.205798485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Crucible ba2ba586-d262-4e31-9136-813d23c6ea4b has session id: 567137ac-3e04-4231-a6e2-67c05176904b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206120387Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Upstairs opts: Upstairs UUID: ba2ba586-d262-4e31-9136-813d23c6ea4b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206150134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Crucible stats registered with UUID: ba2ba586-d262-4e31-9136-813d23c6ea4b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206160643Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:22.206176072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206302381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206332768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206345557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206480275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206499184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"ba2ba586-d262-4e31-9136-813d23c6ea4b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206523381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206532581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.20654218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206551729Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206561058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206570717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206580357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"ds_connection connected from Ok(127.0.0.1:52996)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206635002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"ds_connection connected from Ok(127.0.0.1:35548)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.20666011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"ds_connection connected from Ok(127.0.0.1:51156)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.206719004Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207298544Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.20734624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207369488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207635655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207673311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[0]R flush_numbers[0..12]: [1, 8, 6, 7, 5, 5, 8, 7, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207727927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[0]R generation[0..12]: [1, 5, 4, 4, 3, 3, 5, 4, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207738436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207749175Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[1]R flush_numbers[0..12]: [1, 8, 6, 7, 5, 5, 8, 7, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207759254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[1]R generation[0..12]: [1, 5, 4, 4, 3, 3, 5, 4, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207769173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207779762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[2]R flush_numbers[0..12]: [1, 8, 6, 7, 5, 5, 8, 7, 1, 4, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207789631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[2]R generation[0..12]: [1, 5, 4, 4, 3, 3, 5, 4, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.20779952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207810189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207819319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207828838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Next flush: 9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207837927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207848366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"":"downstairs","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207857845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207866354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"0","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207875774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"1","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207885243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"client":"2","":"downstairs","session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207894992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207903811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"ba2ba586-d262-4e31-9136-813d23c6ea4b is now active with session: 567137ac-3e04-4231-a6e2-67c05176904b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.2079136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.20792241Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"session_id":"567137ac-3e04-4231-a6e2-67c05176904b"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207931339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Activated sub_volume ba2ba586-d262-4e31-9136-813d23c6ea4b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207939138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.207946887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.208282798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} 01/30 Flush 02/30 Write block 51 len 36864 data: 3 3 3 4 3 2 2 2 2 03/30 Write block 232 len 40960 data: 2 2 2 2 2 2 2 2 2 2 04/30 Write block 101 len 24576 data: 3 3 3 3 3 2 05/30 Write block 559 len 16384 data: 2 2 2 2 06/30 Write block 398 len 20480 data: 3 3 3 3 3 07/30 Read block 64 len 4096 08/30 Read block 419 len 4096 09/30 Write block 46 len 4096 data: 2 10/30 Write block 507 len 36864 data: 3 3 3 2 2 2 2 2 2 11/30 Read block 595 len 16384 12/30 Write block 153 len 12288 data: 2 2 2 13/30 Read block 69 len 20480 14/30 Read block 461 len 20480 15/30 Write block 340 len 16384 data: 2 2 2 2 16/30 Write block 552 len 24576 data: 2 2 2 2 2 2 17/30 Read block 512 len 12288 18/30 Flush 19/30 Write block 270 len 12288 data: 2 2 2 20/30 Read block 355 len 40960 21/30 Read block 531 len 24576 22/30 Write block 388 len 24576 data: 2 2 2 2 2 2 23/30 Read block 339 len 24576 24/30 Write block 205 len 8192 data: 2 2 25/30 Read block 201 len 12288 26/30 Read block 560 len 24576 27/30 Read block 371 len 20480 28/30 Read block 381 len 12288 29/30 Write block 421 len 40960 data: 2 2 2 2 2 2 2 2 2 2 30/30 Read block 522 len 16384 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 9 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 10 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 25 26 Done 29 28 4 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:22.216356655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:22.21652693Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1157,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 5 5 5 8 8 8 F F F 2 040-059 6 6 4 10 10 6 F F T <--- 3 060-079 4 4 4 7 7 7 F F F 4 080-099 3 3 3 5 5 5 F F F 5 100-119 6 6 3 10 10 5 F F T <--- 6 120-139 5 5 5 8 8 8 F F F 7 140-159 6 6 4 10 10 7 F F F <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 1 1 1 1 1 1 T T F <--- 11 220-239 6 6 1 10 10 1 F F T <--- 12 240-259 6 6 5 10 10 8 F F T <--- 13 260-279 2 2 2 2 2 2 T T F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 4 4 4 6 6 6 F F F 16 320-339 4 4 4 6 6 6 F F F 17 340-359 6 6 3 10 10 5 F F F <--- 18 360-379 2 2 2 3 3 3 F F F 19 380-399 6 6 4 10 10 6 T T F <--- 20 400-419 6 6 4 10 10 6 F F F <--- 21 420-439 4 4 4 6 6 6 T T F <--- 22 440-459 4 4 4 6 6 6 F F F 23 460-479 2 2 2 4 4 4 F F F 24 480-499 4 4 4 7 7 7 F F F 25 500-519 6 6 2 10 10 4 F F F <--- 26 520-539 3 3 3 5 5 5 F F F 27 540-559 6 6 1 10 10 1 F F T <--- 28 560-579 6 6 1 10 10 1 F F T <--- 29 580-599 4 4 4 7 7 7 F F F Max gen: 6, Max flush: 10 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 5 5 5 8 8 8 F F F 2 040-059 6 6 6 10 10 10 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 3 3 3 5 5 5 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 6 6 6 10 10 10 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 7 7 7 11 11 11 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 4 4 4 6 6 6 F F F 16 320-339 4 4 4 6 6 6 F F F 17 340-359 6 6 6 10 10 10 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 7 7 7 11 11 11 F F F 20 400-419 6 6 6 10 10 10 F F F 21 420-439 7 7 7 11 11 11 F F F 22 440-459 4 4 4 6 6 6 F F F 23 460-479 2 2 2 4 4 4 F F F 24 480-499 4 4 4 7 7 7 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 3 3 3 5 5 5 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 6 6 6 10 10 10 F F F 29 580-599 4 4 4 7 7 7 F F F Max gen: 7, Max flush: 11 Begin loop 4 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.463697654Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:26.470936993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.470990908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.477852171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478118487Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478154444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478163653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478175392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Crucible 035ca6d0-c7e6-4d9f-b5cd-097e486c2e70 has session id: cba67431-e741-42f4-810d-88af7f6bed8f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478467367Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Upstairs opts: Upstairs UUID: 035ca6d0-c7e6-4d9f-b5cd-097e486c2e70, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478489785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Crucible stats registered with UUID: 035ca6d0-c7e6-4d9f-b5cd-097e486c2e70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478499614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:26.478508583Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478637692Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478668219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478678769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478799858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478815497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"035ca6d0-c7e6-4d9f-b5cd-097e486c2e70 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478841244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478850544Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478861873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478871402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478880691Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.47889039Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478900169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58702)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478927837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:49352)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.478948555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35232)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.47901033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.479657723Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.479682791Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.47969442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.479978415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480023491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[0]R flush_numbers[0..12]: [1, 8, 10, 7, 5, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480059708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[0]R generation[0..12]: [1, 5, 6, 4, 3, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480070307Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480083676Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[1]R flush_numbers[0..12]: [1, 8, 10, 7, 5, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480094005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[1]R generation[0..12]: [1, 5, 6, 4, 3, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480103784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480114493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[2]R flush_numbers[0..12]: [1, 8, 10, 7, 5, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480124533Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[2]R generation[0..12]: [1, 5, 6, 4, 3, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480134402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480145031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.48015434Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480163789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Next flush: 12","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480173018Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480182188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"":"downstairs","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480191747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480200266Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"0","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480209685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"1","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480219034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"client":"2","":"downstairs","session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480229753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480238503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"035ca6d0-c7e6-4d9f-b5cd-097e486c2e70 is now active with session: cba67431-e741-42f4-810d-88af7f6bed8f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480248242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480256701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"session_id":"cba67431-e741-42f4-810d-88af7f6bed8f"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.48026551Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Activated sub_volume 035ca6d0-c7e6-4d9f-b5cd-097e486c2e70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.48027326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480281139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.480566344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} 01/30 Write block 561 len 12288 data: 3 3 2 02/30 Write block 12 len 32768 data: 2 2 2 2 2 2 2 2 03/30 Write block 524 len 4096 data: 3 04/30 Read block 533 len 36864 05/30 Read block 362 len 28672 06/30 Write block 280 len 36864 data: 2 2 2 2 2 2 2 2 2 07/30 Flush 08/30 Write block 375 len 32768 data: 2 2 2 2 2 2 2 2 09/30 Read block 436 len 12288 10/30 Read block 83 len 32768 11/30 Read block 270 len 36864 12/30 Read block 227 len 12288 13/30 Read block 94 len 12288 14/30 Write block 536 len 16384 data: 2 2 2 2 15/30 Read block 300 len 12288 16/30 Flush 17/30 Write block 481 len 16384 data: 4 3 3 3 18/30 Write block 474 len 24576 data: 3 3 3 3 3 3 19/30 Write block 381 len 32768 data: 3 3 2 2 2 2 2 3 20/30 Read block 205 len 24576 21/30 Write block 335 len 8192 data: 2 2 22/30 Read block 319 len 8192 23/30 Read block 499 len 4096 24/30 Write block 88 len 28672 data: 2 2 2 2 2 2 2 25/30 Read block 318 len 40960 26/30 Write block 361 len 28672 data: 3 3 3 3 3 3 3 27/30 Write block 349 len 16384 data: 2 3 2 2 28/30 Read block 370 len 20480 29/30 Read block 596 len 16384 30/30 Write block 400 len 24576 data: 4 4 4 3 3 3 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 7 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 1 1 25 Done 0 22 22 44 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:26.488696746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:26.488859192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1165,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 12 12 12 F F F 1 020-039 5 5 5 8 8 8 F F F 2 040-059 6 6 6 10 10 10 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 3 3 3 5 5 5 F T T <--- 5 100-119 6 6 6 10 10 10 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 6 6 6 10 10 10 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 7 7 7 11 11 11 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 4 4 4 6 6 6 F F F 16 320-339 4 4 4 6 6 6 F T T <--- 17 340-359 6 6 6 10 10 10 F T T <--- 18 360-379 2 8 8 3 13 13 F T T <--- 19 380-399 7 8 8 11 13 13 F T T <--- 20 400-419 6 6 6 10 10 10 F T T <--- 21 420-439 7 7 7 11 11 11 F F F 22 440-459 4 4 4 6 6 6 F F F 23 460-479 2 2 2 4 4 4 F T T <--- 24 480-499 4 4 4 7 7 7 F T T <--- 25 500-519 6 6 6 10 10 10 F F F 26 520-539 8 8 8 12 13 13 F F F <--- 27 540-559 6 6 6 10 10 10 F F F 28 560-579 8 8 8 12 12 12 F F F 29 580-599 4 4 4 7 7 7 F F F Max gen: 8, Max flush: 13 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 12 12 12 F F F 1 020-039 5 5 5 8 8 8 F F F 2 040-059 6 6 6 10 10 10 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 6 6 6 10 10 10 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 4 4 4 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 7 7 7 11 11 11 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 4 4 4 6 6 6 F F F 16 320-339 9 9 9 14 14 14 F F F 17 340-359 9 9 9 14 14 14 F F F 18 360-379 9 9 9 14 14 14 F F F 19 380-399 9 9 9 14 14 14 F F F 20 400-419 9 9 9 14 14 14 F F F 21 420-439 7 7 7 11 11 11 F F F 22 440-459 4 4 4 6 6 6 F F F 23 460-479 9 9 9 14 14 14 F F F 24 480-499 9 9 9 14 14 14 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 8 8 8 13 13 13 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 8 8 8 12 12 12 F F F 29 580-599 4 4 4 7 7 7 F F F Max gen: 9, Max flush: 14 Begin loop 5 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.735391251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:30.742538818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.742587264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.749441437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.749696405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.749719153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.749727542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.749738871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Crucible dfc8cfd1-f8b9-400b-8905-d05da368fd9b has session id: 4aacdb13-f39b-4685-96f1-5a33c9644d91","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750080231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Upstairs opts: Upstairs UUID: dfc8cfd1-f8b9-400b-8905-d05da368fd9b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750109099Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Crucible stats registered with UUID: dfc8cfd1-f8b9-400b-8905-d05da368fd9b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750122208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:30.750131517Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750243887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750261615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750271445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750432711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750448319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"dfc8cfd1-f8b9-400b-8905-d05da368fd9b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750479966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750489356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750498775Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750508244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750517713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750527502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750537271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"ds_connection connected from Ok(127.0.0.1:57351)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750575678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"ds_connection connected from Ok(127.0.0.1:52191)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750603066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"ds_connection connected from Ok(127.0.0.1:44564)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.750718226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751252049Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751292596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751313374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751549623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751560592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[0]R flush_numbers[0..12]: [12, 8, 10, 7, 14, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.75159205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[0]R generation[0..12]: [8, 5, 6, 4, 9, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751602989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751613808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[1]R flush_numbers[0..12]: [12, 8, 10, 7, 14, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751623977Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[1]R generation[0..12]: [8, 5, 6, 4, 9, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751633896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751644495Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[2]R flush_numbers[0..12]: [12, 8, 10, 7, 14, 10, 8, 10, 1, 4, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751654534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[2]R generation[0..12]: [8, 5, 6, 4, 9, 6, 5, 6, 1, 2, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751667223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751677892Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751687091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.75169658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Next flush: 15","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.75170571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751714819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"":"downstairs","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751724308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751733017Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"0","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751742356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"1","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751751766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"client":"2","":"downstairs","session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751761405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751770324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"dfc8cfd1-f8b9-400b-8905-d05da368fd9b is now active with session: 4aacdb13-f39b-4685-96f1-5a33c9644d91","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751779863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751788462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"session_id":"4aacdb13-f39b-4685-96f1-5a33c9644d91"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751797382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Activated sub_volume dfc8cfd1-f8b9-400b-8905-d05da368fd9b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.751805431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.75181335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.752175209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} 01/30 Write block 278 len 8192 data: 2 2 02/30 Write block 482 len 40960 data: 4 4 4 2 2 2 2 2 2 2 03/30 Write block 31 len 28672 data: 3 3 3 3 3 3 3 04/30 Write block 47 len 32768 data: 2 3 3 3 4 4 4 5 05/30 Read block 312 len 28672 06/30 Write block 433 len 24576 data: 2 2 2 2 3 3 07/30 Write block 460 len 40960 data: 2 2 2 2 2 2 2 2 2 2 08/30 Read block 339 len 8192 09/30 Write block 353 len 40960 data: 2 2 2 2 2 2 2 3 4 4 10/30 Read block 285 len 36864 11/30 Write block 584 len 12288 data: 3 3 4 12/30 Flush 13/30 Write block 177 len 32768 data: 2 2 2 2 2 2 3 3 14/30 Read block 205 len 20480 15/30 Write block 561 len 36864 data: 4 4 3 2 2 2 2 2 2 16/30 Read block 68 len 12288 17/30 Read block 336 len 40960 18/30 Flush 19/30 Write block 292 len 32768 data: 2 2 2 2 2 2 2 2 20/30 Write block 436 len 20480 data: 3 4 4 3 3 21/30 Read block 209 len 16384 22/30 Write block 278 len 28672 data: 3 3 3 3 3 3 3 23/30 Read block 242 len 28672 24/30 Write block 318 len 28672 data: 3 3 3 3 3 3 3 25/30 Read block 97 len 28672 26/30 Read block 340 len 24576 27/30 Write block 420 len 4096 data: 2 28/30 Write block 448 len 32768 data: 2 2 2 2 2 2 2 2 29/30 Write block 147 len 24576 data: 2 2 2 2 2 2 30/30 Write block 474 len 24576 data: 4 4 4 4 4 4 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 3 3 32 Done 4 27 27 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:30.761024058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:30.761170925Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1173,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 12 12 12 F F F 1 020-039 5 10 10 8 15 15 T F F <--- 2 040-059 6 10 10 10 15 15 T F F <--- 3 060-079 4 4 4 7 7 7 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 6 6 6 10 10 10 F T T <--- 8 160-179 1 10 10 1 16 16 F F F <--- 9 180-199 2 10 10 4 16 16 F F F <--- 10 200-219 7 7 7 11 11 11 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 7 10 10 11 15 15 T T T <--- 14 280-299 8 8 8 12 12 12 F T T <--- 15 300-319 4 4 4 6 6 6 F T T <--- 16 320-339 9 9 9 14 14 14 F T T <--- 17 340-359 9 10 10 14 15 15 F F F <--- 18 360-379 9 10 10 14 15 15 F F F <--- 19 380-399 9 9 9 14 14 14 F F F 20 400-419 9 9 9 14 14 14 F F F 21 420-439 7 10 10 11 15 15 F T T <--- 22 440-459 4 4 4 6 6 6 F T T <--- 23 460-479 9 10 10 14 15 15 F T T <--- 24 480-499 9 10 10 14 15 15 T F F <--- 25 500-519 6 6 6 10 10 10 F F F 26 520-539 8 8 8 13 13 13 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 8 10 10 12 16 16 F F F <--- 29 580-599 4 10 10 7 15 15 F F F <--- Max gen: 10, Max flush: 16 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 12 12 12 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 10 10 10 15 15 15 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 5 5 5 8 8 8 F F F 7 140-159 11 11 11 17 17 17 F F F 8 160-179 10 10 10 16 16 16 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 11 11 11 17 17 17 F F F 15 300-319 11 11 11 17 17 17 F F F 16 320-339 11 11 11 17 17 17 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 9 9 9 14 14 14 F F F 20 400-419 9 9 9 14 14 14 F F F 21 420-439 11 11 11 17 17 17 F F F 22 440-459 11 11 11 17 17 17 F F F 23 460-479 11 11 11 17 17 17 F F F 24 480-499 10 10 10 15 15 15 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 8 8 8 13 13 13 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 10 10 10 15 15 15 F F F Max gen: 11, Max flush: 17 Begin loop 6 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.026010259Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034288408Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034570414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034595781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034604021Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.03461554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Crucible 2e5969d9-6766-4b7f-b651-d7451029074d has session id: 559c9060-41fe-4181-a5b2-3ce4f0225717","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034936222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Upstairs opts: Upstairs UUID: 2e5969d9-6766-4b7f-b651-d7451029074d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.03495532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Crucible stats registered with UUID: 2e5969d9-6766-4b7f-b651-d7451029074d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.034965139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:35.034974138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035167351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.03518367Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035195729Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035347926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035360205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"2e5969d9-6766-4b7f-b651-d7451029074d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035377573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035386182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035395672Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035405041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.03541428Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035423939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035433648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"ds_connection connected from Ok(127.0.0.1:39939)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035480554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"ds_connection connected from Ok(127.0.0.1:42766)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035491243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"ds_connection connected from Ok(127.0.0.1:60106)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.035501342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036173034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036198882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036248407Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036482207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036516184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[0]R flush_numbers[0..12]: [12, 15, 15, 7, 14, 10, 8, 17, 16, 16, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036580188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[0]R generation[0..12]: [8, 10, 10, 4, 9, 6, 5, 11, 10, 10, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036590837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036601797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[1]R flush_numbers[0..12]: [12, 15, 15, 7, 14, 10, 8, 17, 16, 16, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036611956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[1]R generation[0..12]: [8, 10, 10, 4, 9, 6, 5, 11, 10, 10, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036622045Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036633194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[2]R flush_numbers[0..12]: [12, 15, 15, 7, 14, 10, 8, 17, 16, 16, 11, 10]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036643413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[2]R generation[0..12]: [8, 10, 10, 4, 9, 6, 5, 11, 10, 10, 7, 6]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036653532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036664151Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.03667335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036683139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036692359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036701588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"":"downstairs","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036712567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036721196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"0","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036730595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"1","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036739844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"client":"2","":"downstairs","session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036749434Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036758453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"2e5969d9-6766-4b7f-b651-d7451029074d is now active with session: 559c9060-41fe-4181-a5b2-3ce4f0225717","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036768262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036776761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"session_id":"559c9060-41fe-4181-a5b2-3ce4f0225717"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036785571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Activated sub_volume 2e5969d9-6766-4b7f-b651-d7451029074d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.0367937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.036802179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.037149799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} 01/30 Write block 144 len 8192 data: 3 3 02/30 Read block 110 len 24576 03/30 Flush 04/30 Write block 139 len 16384 data: 2 3 3 3 05/30 Flush 06/30 Write block 462 len 36864 data: 3 3 3 3 3 3 3 3 2 07/30 Write block 589 len 32768 data: 5 5 5 4 2 2 2 2 08/30 Read block 92 len 20480 09/30 Write block 227 len 32768 data: 2 2 2 2 2 3 3 3 10/30 Read block 104 len 20480 11/30 Flush 12/30 Write block 207 len 24576 data: 2 2 2 2 2 2 13/30 Write block 430 len 16384 data: 3 2 2 3 14/30 Read block 338 len 36864 15/30 Write block 439 len 40960 data: 4 4 3 3 3 3 2 2 2 3 16/30 Flush 17/30 Write block 317 len 24576 data: 3 4 4 4 4 4 18/30 Read block 101 len 20480 19/30 Write block 235 len 4096 data: 3 20/30 Read block 213 len 24576 21/30 Write block 235 len 12288 data: 4 3 3 22/30 Write block 535 len 8192 data: 2 3 23/30 Write block 170 len 8192 data: 2 2 24/30 Read block 234 len 32768 25/30 Write block 4 len 36864 data: 2 2 2 2 2 2 2 2 3 26/30 Write block 151 len 36864 data: 3 3 3 3 3 2 2 2 2 27/30 Write block 383 len 40960 data: 3 3 3 3 3 4 3 3 3 3 28/30 Read block 505 len 24576 29/30 Write block 299 len 32768 data: 3 2 2 2 2 2 2 2 30/30 Read block 433 len 16384 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 4 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 8 Done Done Sent false 1029 Acked Read 4 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:35.046306741Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:35.046473606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1181,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 12 12 12 T T F <--- 1 020-039 10 10 10 15 15 15 F F F 2 040-059 10 10 10 15 15 15 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 12 12 5 19 19 8 F F F <--- 7 140-159 12 12 12 19 19 18 T T F <--- 8 160-179 10 10 10 16 16 16 T T F <--- 9 180-199 10 10 10 16 16 16 F F F 10 200-219 12 12 7 21 21 11 F F F <--- 11 220-239 12 12 6 20 20 10 T T F <--- 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 11 11 11 17 17 17 T T F <--- 15 300-319 11 11 11 17 17 17 T T F <--- 16 320-339 11 11 11 17 17 17 T T F <--- 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 9 9 9 14 14 14 T T F <--- 20 400-419 9 9 9 14 14 14 F F F 21 420-439 12 12 11 21 21 17 F F F <--- 22 440-459 12 12 11 21 21 17 F F F <--- 23 460-479 12 12 11 20 20 17 F F F <--- 24 480-499 10 10 10 15 15 15 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 8 8 8 13 13 13 T T F <--- 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 12 12 10 20 20 15 F F F <--- Max gen: 12, Max flush: 21 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 10 10 10 15 15 15 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 9 9 9 14 14 14 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 13 13 13 22 22 22 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 12 12 12 21 21 21 F F F 11 220-239 13 13 13 22 22 22 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 13 13 13 22 22 22 F F F 16 320-339 13 13 13 22 22 22 F F F 17 340-359 10 10 10 15 15 15 F F F 18 360-379 10 10 10 15 15 15 F F F 19 380-399 13 13 13 22 22 22 F F F 20 400-419 9 9 9 14 14 14 F F F 21 420-439 12 12 12 21 21 21 F F F 22 440-459 12 12 12 21 21 21 F F F 23 460-479 12 12 12 20 20 20 F F F 24 480-499 10 10 10 15 15 15 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 13 13 13 22 22 22 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 12 12 12 20 20 20 F F F Max gen: 13, Max flush: 22 Begin loop 7 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.30236902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.310565776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.310899767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.310926215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.310937814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.310949823Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Crucible 82c34dc0-252c-45e3-9b9b-1c75f50b10c9 has session id: 3874a5f9-7920-48b6-a66a-87df619d5b36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311258616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Upstairs opts: Upstairs UUID: 82c34dc0-252c-45e3-9b9b-1c75f50b10c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311287343Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Crucible stats registered with UUID: 82c34dc0-252c-45e3-9b9b-1c75f50b10c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311297902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:39.311308191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311474067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311503614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311514033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311661961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311679689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"82c34dc0-252c-45e3-9b9b-1c75f50b10c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311712336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311721625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311731324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311740744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311750033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311762022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311771641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"ds_connection connected from Ok(127.0.0.1:65407)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311938956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"ds_connection connected from Ok(127.0.0.1:46754)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311963984Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"ds_connection connected from Ok(127.0.0.1:59377)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.311980173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.31247306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312502517Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312514416Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.31281786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312827149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 15, 7, 14, 10, 19, 22, 22, 16, 21, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312846487Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[0]R generation[0..12]: [13, 10, 10, 4, 9, 6, 12, 13, 13, 10, 12, 13]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312856876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312867695Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 15, 7, 14, 10, 19, 22, 22, 16, 21, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312877935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[1]R generation[0..12]: [13, 10, 10, 4, 9, 6, 12, 13, 13, 10, 12, 13]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312890753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312901483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 15, 7, 14, 10, 19, 22, 22, 16, 21, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312911482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[2]R generation[0..12]: [13, 10, 10, 4, 9, 6, 12, 13, 13, 10, 12, 13]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312921491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.31293204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312941329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312950888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Next flush: 23","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312959937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312968997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"":"downstairs","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312978336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312986785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"0","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.312996094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"1","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313005483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"client":"2","":"downstairs","session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313015233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313024212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"82c34dc0-252c-45e3-9b9b-1c75f50b10c9 is now active with session: 3874a5f9-7920-48b6-a66a-87df619d5b36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313033981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.31304385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"session_id":"3874a5f9-7920-48b6-a66a-87df619d5b36"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313052809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Activated sub_volume 82c34dc0-252c-45e3-9b9b-1c75f50b10c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313060709Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313068588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.313412858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} 01/30 Write block 153 len 16384 data: 4 4 4 3 02/30 Read block 266 len 8192 03/30 Write block 80 len 12288 data: 2 2 2 04/30 Read block 383 len 24576 05/30 Write block 338 len 12288 data: 2 2 3 06/30 Write block 486 len 40960 data: 3 3 3 3 3 3 3 3 3 3 07/30 Read block 217 len 16384 08/30 Write block 446 len 20480 data: 3 3 4 3 3 09/30 Flush 10/30 Write block 97 len 4096 data: 2 11/30 Read block 351 len 8192 12/30 Read block 36 len 28672 13/30 Read block 148 len 32768 14/30 Read block 113 len 20480 15/30 Write block 580 len 28672 data: 2 2 2 3 4 4 5 16/30 Read block 221 len 36864 17/30 Write block 442 len 40960 data: 4 4 4 3 4 4 5 4 4 3 18/30 Read block 215 len 24576 19/30 Read block 94 len 16384 20/30 Read block 313 len 4096 21/30 Write block 401 len 20480 data: 5 5 4 4 4 22/30 Read block 211 len 12288 23/30 Read block 358 len 12288 24/30 Write block 219 len 8192 data: 2 2 25/30 Write block 58 len 24576 data: 3 3 3 3 3 3 26/30 Write block 376 len 12288 data: 3 3 3 27/30 Read block 165 len 12288 28/30 Read block 592 len 4096 29/30 Read block 406 len 36864 30/30 Write block 364 len 32768 data: 4 4 4 4 3 3 2 2 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 3 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 29 29 4 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) Some(JobId(1008)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:39.321498893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:39.321689167Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1189,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 10 10 10 15 15 15 T T F <--- 3 060-079 4 4 4 7 7 7 T T F <--- 4 080-099 14 14 9 23 23 14 T T T <--- 5 100-119 6 6 6 10 10 10 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 14 14 13 23 23 22 F F T <--- 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 12 12 12 21 21 21 T T F <--- 11 220-239 13 13 13 22 22 22 T T F <--- 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 13 13 13 22 22 22 F F F 16 320-339 14 14 13 23 23 22 F F F <--- 17 340-359 14 14 10 23 23 15 F F F <--- 18 360-379 10 10 10 15 15 15 T T F <--- 19 380-399 13 13 13 22 22 22 F F F 20 400-419 9 9 9 14 14 14 T T F <--- 21 420-439 12 12 12 21 21 21 F F F 22 440-459 14 14 12 23 23 21 T T F <--- 23 460-479 12 12 12 20 20 20 F F F 24 480-499 14 14 10 23 23 15 F F F <--- 25 500-519 6 6 6 10 10 10 F F F 26 520-539 13 13 13 22 22 22 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 12 12 12 20 20 20 T T F <--- Max gen: 14, Max flush: 23 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 15 15 15 24 24 24 F F F 3 060-079 15 15 15 24 24 24 F F F 4 080-099 15 15 15 24 24 24 F F F 5 100-119 6 6 6 10 10 10 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 14 14 14 23 23 23 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 15 15 15 24 24 24 F F F 11 220-239 15 15 15 24 24 24 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 13 13 13 22 22 22 F F F 16 320-339 14 14 14 23 23 23 F F F 17 340-359 14 14 14 23 23 23 F F F 18 360-379 15 15 15 24 24 24 F F F 19 380-399 13 13 13 22 22 22 F F F 20 400-419 15 15 15 24 24 24 F F F 21 420-439 12 12 12 21 21 21 F F F 22 440-459 15 15 15 24 24 24 F F F 23 460-479 12 12 12 20 20 20 F F F 24 480-499 14 14 14 23 23 23 F F F 25 500-519 6 6 6 10 10 10 F F F 26 520-539 13 13 13 22 22 22 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 15 15 15 24 24 24 F F F Max gen: 15, Max flush: 24 Begin loop 8 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.57150726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:43.578769567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.578835022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586029005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586301591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586328079Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586336458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586347797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Crucible c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9 has session id: a8e71b7c-4557-415b-bed4-fbc9fee4335f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586669609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Upstairs opts: Upstairs UUID: c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586724614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Crucible stats registered with UUID: c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586736973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:43.586745802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.5868866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586903089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.586913118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587053246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587069654Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587086263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587094892Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587104531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.5871141Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.58712352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587133499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587143348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60472)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587207722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"ds_connection connected from Ok(127.0.0.1:59702)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.58723338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"ds_connection connected from Ok(127.0.0.1:53953)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587302164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587867295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.587885183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588014042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588225034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588235523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 24, 24, 24, 10, 19, 23, 22, 16, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588295477Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[0]R generation[0..12]: [13, 10, 15, 15, 15, 6, 12, 14, 13, 10, 15, 15]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588306556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588317446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 24, 24, 24, 10, 19, 23, 22, 16, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588327845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[1]R generation[0..12]: [13, 10, 15, 15, 15, 6, 12, 14, 13, 10, 15, 15]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588337914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588348503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 24, 24, 24, 10, 19, 23, 22, 16, 24, 24]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588358722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[2]R generation[0..12]: [13, 10, 15, 15, 15, 6, 12, 14, 13, 10, 15, 15]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588368821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.58838175Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588391429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588400998Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Next flush: 25","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588410167Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588419497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"":"downstairs","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588429086Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588437745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"0","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588447334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"1","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588456823Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"client":"2","":"downstairs","session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588466353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588475182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9 is now active with session: a8e71b7c-4557-415b-bed4-fbc9fee4335f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588484991Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.58849363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"session_id":"a8e71b7c-4557-415b-bed4-fbc9fee4335f"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588502489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Activated sub_volume c5bf5cff-3af2-41bc-ba5a-835a44e9d2c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588510279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588518148Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.588866498Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} 01/30 Write block 146 len 16384 data: 2 3 3 3 02/30 Write block 374 len 32768 data: 2 3 4 4 4 3 3 4 03/30 Read block 73 len 16384 04/30 Write block 486 len 20480 data: 4 4 4 4 4 05/30 Write block 521 len 36864 data: 2 2 2 4 3 3 3 2 2 06/30 Write block 504 len 16384 data: 3 3 3 4 07/30 Read block 366 len 4096 08/30 Read block 197 len 32768 09/30 Read block 104 len 12288 10/30 Write block 217 len 20480 data: 2 2 3 3 2 11/30 Write block 307 len 16384 data: 2 2 2 2 12/30 Write block 61 len 4096 data: 4 13/30 Write block 118 len 8192 data: 2 2 14/30 Write block 519 len 8192 data: 2 2 15/30 Write block 470 len 36864 data: 3 2 2 2 5 5 5 5 5 16/30 Read block 444 len 24576 17/30 Read block 168 len 20480 18/30 Read block 186 len 36864 19/30 Read block 276 len 4096 20/30 Read block 263 len 4096 21/30 Read block 358 len 12288 22/30 Read block 546 len 8192 23/30 Read block 156 len 28672 24/30 Write block 506 len 36864 data: 4 5 4 4 3 3 3 3 3 25/30 Read block 210 len 32768 26/30 Read block 212 len 36864 27/30 Write block 440 len 20480 data: 5 4 5 5 5 28/30 Read block 101 len 12288 29/30 Read block 183 len 20480 30/30 Read block 468 len 24576 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:43.59641946Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:43.596587735Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1197,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 15 15 15 24 24 24 F F F 3 060-079 15 15 15 24 24 24 F T T <--- 4 080-099 15 15 15 24 24 24 F F F 5 100-119 6 6 6 10 10 10 F T T <--- 6 120-139 12 12 12 19 19 19 F F F 7 140-159 14 14 14 23 23 23 T T T 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 15 15 15 24 24 24 F T T <--- 11 220-239 15 15 15 24 24 24 F T T <--- 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 13 13 13 22 22 22 F T T <--- 16 320-339 14 14 14 23 23 23 F F F 17 340-359 14 14 14 23 23 23 F F F 18 360-379 15 15 15 24 24 24 F T T <--- 19 380-399 13 13 13 22 22 22 F T T <--- 20 400-419 15 15 15 24 24 24 F F F 21 420-439 12 12 12 21 21 21 F F F 22 440-459 15 15 15 24 24 24 F T T <--- 23 460-479 12 12 12 20 20 20 F T T <--- 24 480-499 14 14 14 23 23 23 F T T <--- 25 500-519 6 6 6 10 10 10 F T T <--- 26 520-539 13 13 13 22 22 22 F T T <--- 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 15 15 15 24 24 24 F F F Max gen: 15, Max flush: 24 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 15 15 15 24 24 24 F F F 3 060-079 16 16 16 25 25 25 F F F 4 080-099 15 15 15 24 24 24 F F F 5 100-119 16 16 16 25 25 25 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 16 16 16 25 25 25 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 16 16 16 25 25 25 F F F 11 220-239 16 16 16 25 25 25 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 13 13 13 22 22 22 F F F 15 300-319 16 16 16 25 25 25 F F F 16 320-339 14 14 14 23 23 23 F F F 17 340-359 14 14 14 23 23 23 F F F 18 360-379 16 16 16 25 25 25 F F F 19 380-399 16 16 16 25 25 25 F F F 20 400-419 15 15 15 24 24 24 F F F 21 420-439 12 12 12 21 21 21 F F F 22 440-459 16 16 16 25 25 25 F F F 23 460-479 16 16 16 25 25 25 F F F 24 480-499 16 16 16 25 25 25 F F F 25 500-519 16 16 16 25 25 25 F F F 26 520-539 16 16 16 25 25 25 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 15 15 15 24 24 24 F F F Max gen: 16, Max flush: 25 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.852570211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.860739999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861031394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861056732Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861065211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.86107655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Crucible 02c91292-76cf-41f6-bc45-209f51cd385e has session id: 2f266837-7395-4226-ba22-4be32610947d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861410901Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Upstairs opts: Upstairs UUID: 02c91292-76cf-41f6-bc45-209f51cd385e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861430919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Crucible stats registered with UUID: 02c91292-76cf-41f6-bc45-209f51cd385e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861440838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:47.861449657Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861597305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861628232Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861638891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.86176261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861778439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"02c91292-76cf-41f6-bc45-209f51cd385e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861814706Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861823975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861833484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861843023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861852482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861862172Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861871741Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"ds_connection connected from Ok(127.0.0.1:60893)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.861964453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"ds_connection connected from Ok(127.0.0.1:55141)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.86198956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"ds_connection connected from Ok(127.0.0.1:65326)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862000939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862614686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862640224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862675481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862925639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.862960766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 24, 25, 24, 25, 19, 25, 22, 16, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863012681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[0]R generation[0..12]: [13, 10, 15, 16, 15, 16, 12, 16, 13, 10, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.86302355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863034689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 24, 25, 24, 25, 19, 25, 22, 16, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863044938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[1]R generation[0..12]: [13, 10, 15, 16, 15, 16, 12, 16, 13, 10, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863055048Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863065757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 24, 25, 24, 25, 19, 25, 22, 16, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863076136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[2]R generation[0..12]: [13, 10, 15, 16, 15, 16, 12, 16, 13, 10, 16, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863086235Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863096844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Max found gen is 17","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863105923Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Generation requested: 18 >= found:17","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863115472Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Next flush: 26","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863124502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863133571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"":"downstairs","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.86314314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863151639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"0","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863162458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"1","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863171867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"client":"2","":"downstairs","session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863181577Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863190486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"02c91292-76cf-41f6-bc45-209f51cd385e is now active with session: 2f266837-7395-4226-ba22-4be32610947d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863200205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863208724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"session_id":"2f266837-7395-4226-ba22-4be32610947d"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863217603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Activated sub_volume 02c91292-76cf-41f6-bc45-209f51cd385e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863225623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863233652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.863534536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} 01/30 Write block 446 len 20480 data: 5 5 6 5 5 02/30 Flush 03/30 Write block 584 len 12288 data: 5 5 6 04/30 Read block 549 len 12288 05/30 Read block 66 len 36864 06/30 Read block 52 len 32768 07/30 Write block 59 len 24576 data: 4 4 5 4 4 3 08/30 Read block 570 len 36864 09/30 Flush 10/30 Write block 515 len 32768 data: 3 2 2 2 3 3 3 3 11/30 Read block 512 len 40960 12/30 Read block 509 len 24576 13/30 Read block 151 len 32768 14/30 Write block 293 len 32768 data: 3 3 3 3 3 3 4 3 15/30 Read block 370 len 36864 16/30 Read block 124 len 20480 17/30 Read block 164 len 24576 18/30 Flush 19/30 Write block 394 len 20480 data: 2 2 2 2 4 20/30 Write block 397 len 8192 data: 3 5 21/30 Write block 202 len 20480 data: 2 2 2 3 3 22/30 Write block 354 len 24576 data: 3 3 3 3 3 3 23/30 Write block 76 len 12288 data: 2 2 2 24/30 Read block 255 len 8192 25/30 Write block 425 len 24576 data: 3 3 3 3 3 4 26/30 Write block 295 len 40960 data: 4 4 4 4 5 4 3 3 3 3 27/30 Read block 588 len 36864 28/30 Read block 19 len 4096 29/30 Write block 354 len 4096 data: 4 30/30 Read block 37 len 40960 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:47.872753383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:47.872926878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1205,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 18 15 18 27 24 27 F F F <--- 3 060-079 18 16 18 27 25 27 T F T <--- 4 080-099 15 15 15 24 24 24 F F F 5 100-119 16 16 16 25 25 25 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 16 16 16 25 25 25 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 16 16 16 25 25 25 T F T <--- 11 220-239 16 16 16 25 25 25 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 18 13 18 28 22 28 T F T <--- 15 300-319 18 16 18 28 25 28 T F T <--- 16 320-339 14 14 14 23 23 23 F F F 17 340-359 14 14 14 23 23 23 T F T <--- 18 360-379 16 16 16 25 25 25 F F F 19 380-399 16 16 16 25 25 25 T F T <--- 20 400-419 15 15 15 24 24 24 F F F 21 420-439 12 12 12 21 21 21 T F T <--- 22 440-459 18 16 18 26 25 26 F F F <--- 23 460-479 16 16 16 25 25 25 F F F 24 480-499 16 16 16 25 25 25 F F F 25 500-519 18 16 18 28 25 28 F F F <--- 26 520-539 18 16 18 28 25 28 F F F <--- 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 18 15 18 27 24 27 F F F <--- Max gen: 18, Max flush: 28 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 18 18 18 27 27 27 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 15 15 15 24 24 24 F F F 5 100-119 16 16 16 25 25 25 F F F 6 120-139 12 12 12 19 19 19 F F F 7 140-159 16 16 16 25 25 25 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 19 19 19 29 29 29 F F F 11 220-239 16 16 16 25 25 25 F F F 12 240-259 6 6 6 10 10 10 F F F 13 260-279 11 11 11 17 17 17 F F F 14 280-299 19 19 19 29 29 29 F F F 15 300-319 19 19 19 29 29 29 F F F 16 320-339 14 14 14 23 23 23 F F F 17 340-359 19 19 19 29 29 29 F F F 18 360-379 16 16 16 25 25 25 F F F 19 380-399 19 19 19 29 29 29 F F F 20 400-419 15 15 15 24 24 24 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 18 18 18 26 26 26 F F F 23 460-479 16 16 16 25 25 25 F F F 24 480-499 16 16 16 25 25 25 F F F 25 500-519 18 18 18 28 28 28 F F F 26 520-539 18 18 18 28 28 28 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 10 10 16 16 16 F F F 29 580-599 18 18 18 27 27 27 F F F Max gen: 19, Max flush: 29 Begin loop 10 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.122919744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:52.130117777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.130171662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137169483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13742697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137452068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137460437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137473976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Crucible 56e1d849-34e8-464a-8040-8bab05763cf0 has session id: 94cae6e6-d147-4537-8b47-dd67ab50bec0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137850243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Upstairs opts: Upstairs UUID: 56e1d849-34e8-464a-8040-8bab05763cf0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.137883901Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Crucible stats registered with UUID: 56e1d849-34e8-464a-8040-8bab05763cf0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13789563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:52.137905979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138015369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138033857Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138044787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138197443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138211872Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"56e1d849-34e8-464a-8040-8bab05763cf0 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138227621Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13823744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138249929Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138268157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138278086Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138290785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138301324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"ds_connection connected from Ok(127.0.0.1:41929)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138328432Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"ds_connection connected from Ok(127.0.0.1:40038)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138383207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"ds_connection connected from Ok(127.0.0.1:41066)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.138407915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139044729Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139069277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139080846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139357222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13938177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 27, 29, 24, 25, 19, 25, 22, 16, 29, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139469532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[0]R generation[0..12]: [13, 10, 18, 19, 15, 16, 12, 16, 13, 10, 19, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139480491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13949152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 27, 29, 24, 25, 19, 25, 22, 16, 29, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13950185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[1]R generation[0..12]: [13, 10, 18, 19, 15, 16, 12, 16, 13, 10, 19, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139514149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139524928Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 27, 29, 24, 25, 19, 25, 22, 16, 29, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139535077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[2]R generation[0..12]: [13, 10, 18, 19, 15, 16, 12, 16, 13, 10, 19, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139545326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139556015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139565274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139574913Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139584122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139593292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"":"downstairs","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139602881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.13961157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"0","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139621109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"1","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139630548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"client":"2","":"downstairs","session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139640228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139649207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"56e1d849-34e8-464a-8040-8bab05763cf0 is now active with session: 94cae6e6-d147-4537-8b47-dd67ab50bec0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139658966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139668805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"session_id":"94cae6e6-d147-4537-8b47-dd67ab50bec0"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139677754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Activated sub_volume 56e1d849-34e8-464a-8040-8bab05763cf0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139685674Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139693723Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.139996377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} 01/30 Write block 113 len 40960 data: 2 2 2 2 2 3 3 2 2 2 02/30 Write block 512 len 36864 data: 4 4 4 4 3 3 3 4 4 03/30 Write block 278 len 12288 data: 4 4 4 04/30 Read block 437 len 28672 05/30 Write block 47 len 8192 data: 3 4 06/30 Write block 248 len 12288 data: 3 3 3 07/30 Write block 398 len 28672 data: 6 4 5 6 6 5 5 08/30 Read block 150 len 4096 09/30 Write block 569 len 4096 data: 3 10/30 Read block 498 len 4096 11/30 Write block 122 len 24576 data: 3 2 2 2 2 2 12/30 Write block 261 len 8192 data: 4 3 13/30 Write block 363 len 4096 data: 4 14/30 Read block 448 len 24576 15/30 Flush 16/30 Write block 261 len 12288 data: 5 4 3 17/30 Write block 281 len 28672 data: 4 4 4 4 3 3 3 18/30 Write block 86 len 20480 data: 2 2 3 3 3 19/30 Write block 521 len 4096 data: 4 20/30 Write block 368 len 40960 data: 4 4 3 3 2 2 3 4 5 5 21/30 Read block 236 len 8192 22/30 Read block 580 len 28672 23/30 Read block 332 len 28672 24/30 Read block 123 len 28672 25/30 Read block 116 len 24576 26/30 Read block 167 len 40960 27/30 Read block 62 len 8192 28/30 Write block 211 len 36864 data: 3 3 2 2 2 2 3 3 4 29/30 Read block 424 len 36864 30/30 Read block 443 len 28672 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:52.147884529Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:52.148067953Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1213,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 18 20 20 27 30 30 F F F <--- 3 060-079 19 19 19 29 29 29 F F F 4 080-099 15 15 15 24 24 24 F T T <--- 5 100-119 16 20 20 25 30 30 T F F <--- 6 120-139 12 20 20 19 30 30 T F F <--- 7 140-159 16 16 16 25 25 25 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 19 19 19 29 29 29 F T T <--- 11 220-239 16 16 16 25 25 25 F F F 12 240-259 6 20 20 10 30 30 F F F <--- 13 260-279 11 20 20 17 30 30 F T T <--- 14 280-299 19 20 20 29 30 30 F T T <--- 15 300-319 19 19 19 29 29 29 F F F 16 320-339 14 14 14 23 23 23 F F F 17 340-359 19 19 19 29 29 29 F F F 18 360-379 16 20 20 25 30 30 F T T <--- 19 380-399 19 20 20 29 30 30 F F F <--- 20 400-419 15 20 20 24 30 30 F F F <--- 21 420-439 19 19 19 29 29 29 F F F 22 440-459 18 18 18 26 26 26 F F F 23 460-479 16 16 16 25 25 25 F F F 24 480-499 16 16 16 25 25 25 F F F 25 500-519 18 20 20 28 30 30 F F F <--- 26 520-539 18 20 20 28 30 30 F T T <--- 27 540-559 6 6 6 10 10 10 F F F 28 560-579 10 20 20 16 30 30 F F F <--- 29 580-599 18 18 18 27 27 27 F F F Max gen: 20, Max flush: 30 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 20 20 20 30 30 30 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F F F 5 100-119 20 20 20 30 30 30 F F F 6 120-139 20 20 20 30 30 30 F F F 7 140-159 16 16 16 25 25 25 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 21 21 21 31 31 31 F F F 11 220-239 16 16 16 25 25 25 F F F 12 240-259 20 20 20 30 30 30 F F F 13 260-279 21 21 21 31 31 31 F F F 14 280-299 21 21 21 31 31 31 F F F 15 300-319 19 19 19 29 29 29 F F F 16 320-339 14 14 14 23 23 23 F F F 17 340-359 19 19 19 29 29 29 F F F 18 360-379 21 21 21 31 31 31 F F F 19 380-399 20 20 20 30 30 30 F F F 20 400-419 20 20 20 30 30 30 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 18 18 18 26 26 26 F F F 23 460-479 16 16 16 25 25 25 F F F 24 480-499 16 16 16 25 25 25 F F F 25 500-519 20 20 20 30 30 30 F F F 26 520-539 21 21 21 31 31 31 F F F 27 540-559 6 6 6 10 10 10 F F F 28 560-579 20 20 20 30 30 30 F F F 29 580-599 18 18 18 27 27 27 F F F Max gen: 21, Max flush: 31 Begin loop 11 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.407742308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:56.415100827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.415159991Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422306759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422572776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422600473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422608933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422620512Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Crucible 3b2f66e6-e6c4-4410-ab2b-fe17413548c9 has session id: cc91ff62-638c-4500-be0d-2e730658d8a8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422943673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Upstairs opts: Upstairs UUID: 3b2f66e6-e6c4-4410-ab2b-fe17413548c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422975061Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Crucible stats registered with UUID: 3b2f66e6-e6c4-4410-ab2b-fe17413548c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.422987949Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:56.422997199Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423122898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423139446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423149225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423311181Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.42332762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"3b2f66e6-e6c4-4410-ab2b-fe17413548c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423359927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423369186Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423378655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423388075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423397524Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423407263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423416982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:63781)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423465988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:61776)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423476717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:43962)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.423488326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424145659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424185485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424199454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424438623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424462501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 25, 22, 16, 31, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424556183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[0]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 16, 13, 10, 21, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424567042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424577891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 25, 22, 16, 31, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.42458824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[1]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 16, 13, 10, 21, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424598459Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424609118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 25, 22, 16, 31, 25]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424619427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[2]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 16, 13, 10, 21, 16]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424629576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424642265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424651525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424660884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Next flush: 32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424670123Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424679262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"":"downstairs","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424688641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424697261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"0","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.4247067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"1","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424716449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"client":"2","":"downstairs","session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424726228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424735057Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"3b2f66e6-e6c4-4410-ab2b-fe17413548c9 is now active with session: cc91ff62-638c-4500-be0d-2e730658d8a8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424744746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424753346Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"session_id":"cc91ff62-638c-4500-be0d-2e730658d8a8"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424762305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Activated sub_volume 3b2f66e6-e6c4-4410-ab2b-fe17413548c9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424770664Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.424778823Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.425127893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} 01/30 Write block 303 len 36864 data: 4 4 3 3 3 3 3 3 2 02/30 Write block 203 len 16384 data: 3 3 4 4 03/30 Write block 398 len 40960 data: 7 5 6 7 7 6 6 5 3 3 04/30 Write block 552 len 16384 data: 3 3 3 3 05/30 Write block 157 len 12288 data: 3 3 3 06/30 Flush 07/30 Write block 454 len 40960 data: 3 3 2 2 2 2 3 3 4 4 08/30 Write block 586 len 16384 data: 7 5 5 6 09/30 Flush 10/30 Write block 294 len 40960 data: 4 5 5 5 5 6 5 4 4 5 11/30 Write block 330 len 12288 data: 2 2 2 12/30 Write block 590 len 32768 data: 6 6 5 3 3 3 3 2 13/30 Write block 522 len 4096 data: 4 14/30 Read block 414 len 40960 15/30 Flush 16/30 Write block 207 len 4096 data: 3 17/30 Write block 480 len 36864 data: 3 5 5 5 5 3 5 5 5 18/30 Flush 19/30 Write block 233 len 20480 data: 4 4 5 4 4 20/30 Write block 322 len 12288 data: 5 4 4 21/30 Write block 266 len 36864 data: 3 3 2 2 3 3 3 2 2 22/30 Flush 23/30 Write block 385 len 32768 data: 4 4 4 5 4 4 4 4 24/30 Read block 42 len 20480 25/30 Read block 430 len 12288 26/30 Read block 34 len 20480 27/30 Read block 557 len 40960 28/30 Write block 345 len 36864 data: 2 2 2 2 3 4 3 3 3 29/30 Write block 314 len 16384 data: 2 2 3 4 30/30 Write block 564 len 12288 data: 3 3 3 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 3 3 34 Done 2 27 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:28:56.433236377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:28:56.433417171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1221,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 20 20 20 30 30 30 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F F F 5 100-119 20 20 20 30 30 30 F F F 6 120-139 20 20 20 30 30 30 F F F 7 140-159 16 22 22 25 32 32 F F F <--- 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 21 22 22 31 35 35 T F F <--- 11 220-239 16 22 22 25 36 36 F F F <--- 12 240-259 20 20 20 30 30 30 F F F 13 260-279 21 22 22 31 36 36 F F F <--- 14 280-299 21 22 22 31 34 34 F F F <--- 15 300-319 19 22 22 29 34 34 T T T <--- 16 320-339 14 22 22 23 36 36 F F F <--- 17 340-359 19 19 19 29 29 29 F T T <--- 18 360-379 21 21 21 31 31 31 F F F 19 380-399 20 22 22 30 32 32 F T T <--- 20 400-419 20 22 22 30 32 32 F F F <--- 21 420-439 19 19 19 29 29 29 F F F 22 440-459 18 22 22 26 33 33 F F F <--- 23 460-479 16 22 22 25 33 33 F F F <--- 24 480-499 16 22 22 25 35 35 F F F <--- 25 500-519 20 20 20 30 30 30 F F F 26 520-539 21 22 22 31 34 34 F F F <--- 27 540-559 6 22 22 10 32 32 F F F <--- 28 560-579 20 20 20 30 30 30 F T T <--- 29 580-599 18 22 22 27 34 34 F F F <--- Max gen: 22, Max flush: 36 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 20 20 20 30 30 30 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F F F 5 100-119 20 20 20 30 30 30 F F F 6 120-139 20 20 20 30 30 30 F F F 7 140-159 22 22 22 32 32 32 F F F 8 160-179 13 13 13 22 22 22 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 22 22 22 35 35 35 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 20 20 20 30 30 30 F F F 13 260-279 22 22 22 36 36 36 F F F 14 280-299 22 22 22 34 34 34 F F F 15 300-319 23 23 23 37 37 37 F F F 16 320-339 22 22 22 36 36 36 F F F 17 340-359 23 23 23 37 37 37 F F F 18 360-379 21 21 21 31 31 31 F F F 19 380-399 23 23 23 37 37 37 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 22 22 22 33 33 33 F F F 23 460-479 22 22 22 33 33 33 F F F 24 480-499 22 22 22 35 35 35 F F F 25 500-519 20 20 20 30 30 30 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 23, Max flush: 37 Begin loop 12 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.696746987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705060643Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705345808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705370066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705378555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705389934Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Crucible 682b97f4-4b23-4862-b388-77848380c784 has session id: 4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705752682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Upstairs opts: Upstairs UUID: 682b97f4-4b23-4862-b388-77848380c784, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.70578056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Crucible stats registered with UUID: 682b97f4-4b23-4862-b388-77848380c784","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705791959Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:00.705801238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705927827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705958374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.705969033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706093593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706112361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"682b97f4-4b23-4862-b388-77848380c784 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706138289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706147598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706157077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706166486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706175845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706212772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.70623443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:61578)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706245769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:51683)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706256128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"ds_connection connected from Ok(127.0.0.1:47745)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706377128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.7069238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706941589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.706958807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707214275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707232773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 32, 22, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707284779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[0]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 22, 13, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707295668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707306627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 32, 22, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707316926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[1]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 22, 13, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707327115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707337774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 30, 29, 31, 30, 30, 32, 22, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707347983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[2]R generation[0..12]: [13, 10, 20, 19, 21, 20, 20, 22, 13, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707357992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707368461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707377661Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.70738725Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Next flush: 38","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707396689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707405798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"":"downstairs","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707415337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707425127Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"0","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707434486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"1","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707443895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"client":"2","":"downstairs","session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707453474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707462393Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"682b97f4-4b23-4862-b388-77848380c784 is now active with session: 4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707472192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707480732Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"session_id":"4ada2f06-afb7-4b0d-bd7d-15676c5dc7f8"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707489641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Activated sub_volume 682b97f4-4b23-4862-b388-77848380c784","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.70749781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.70750586Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.707826022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} 01/30 Write block 157 len 8192 data: 4 4 02/30 Write block 130 len 40960 data: 2 2 2 2 2 2 2 2 2 3 03/30 Read block 49 len 16384 04/30 Write block 48 len 8192 data: 5 4 05/30 Write block 322 len 36864 data: 6 5 5 2 2 2 2 2 3 06/30 Write block 120 len 36864 data: 3 3 4 3 3 3 3 3 3 07/30 Flush 08/30 Write block 264 len 4096 data: 3 09/30 Write block 265 len 20480 data: 3 4 4 3 3 10/30 Write block 469 len 32768 data: 4 4 3 3 3 6 6 6 11/30 Read block 570 len 32768 12/30 Read block 205 len 16384 13/30 Read block 555 len 4096 14/30 Write block 357 len 36864 data: 4 4 4 4 5 5 5 5 5 15/30 Read block 280 len 24576 16/30 Read block 590 len 4096 17/30 Read block 17 len 28672 18/30 Flush 19/30 Write block 173 len 4096 data: 2 20/30 Read block 444 len 16384 21/30 Read block 163 len 32768 22/30 Write block 389 len 16384 data: 5 5 5 5 23/30 Read block 353 len 12288 24/30 Read block 588 len 8192 25/30 Write block 164 len 8192 data: 2 2 26/30 Write block 299 len 28672 data: 7 6 5 5 6 5 4 27/30 Read block 541 len 16384 28/30 Write block 244 len 32768 data: 2 2 3 3 4 4 4 3 29/30 Write block 453 len 16384 data: 3 4 4 3 30/30 Read block 557 len 4096 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 26 26 Done 30 30 4 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:00.715752361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:00.715925226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1229,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 24 24 20 38 38 30 F F T <--- 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F F F 5 100-119 20 20 20 30 30 30 F F F 6 120-139 24 24 20 38 38 30 F F T <--- 7 140-159 24 24 22 38 38 32 F F T <--- 8 160-179 13 13 13 22 22 22 T T F <--- 9 180-199 10 10 10 16 16 16 F F F 10 200-219 22 22 22 35 35 35 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 20 20 20 30 30 30 T T F <--- 13 260-279 24 24 22 39 39 36 F F F <--- 14 280-299 22 22 22 34 34 34 T T F <--- 15 300-319 23 23 23 37 37 37 T T F <--- 16 320-339 24 24 22 38 38 36 F F F <--- 17 340-359 24 24 23 39 39 37 F F F <--- 18 360-379 24 24 21 39 39 31 F F F <--- 19 380-399 23 23 23 37 37 37 T T F <--- 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 22 22 22 33 33 33 T T F <--- 23 460-479 24 24 22 39 39 33 F F F <--- 24 480-499 22 22 22 35 35 35 F F F 25 500-519 20 20 20 30 30 30 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 24, Max flush: 39 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 24 24 24 38 38 38 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F F F 5 100-119 20 20 20 30 30 30 F F F 6 120-139 24 24 24 38 38 38 F F F 7 140-159 24 24 24 38 38 38 F F F 8 160-179 25 25 25 40 40 40 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 22 22 22 35 35 35 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 25 25 25 40 40 40 F F F 13 260-279 24 24 24 39 39 39 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 25 25 25 40 40 40 F F F 16 320-339 24 24 24 38 38 38 F F F 17 340-359 24 24 24 39 39 39 F F F 18 360-379 24 24 24 39 39 39 F F F 19 380-399 25 25 25 40 40 40 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 25 25 25 40 40 40 F F F 23 460-479 24 24 24 39 39 39 F F F 24 480-499 22 22 22 35 35 35 F F F 25 500-519 20 20 20 30 30 30 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 25, Max flush: 40 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.965650627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:04.972873038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.972922313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.979798524Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980053772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.98007832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980089219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980100698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Crucible 9017d204-fc9d-4cdd-83b6-2637c9202024 has session id: 6c657847-5beb-4b24-9b12-16f0b1eb2925","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980503793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Upstairs opts: Upstairs UUID: 9017d204-fc9d-4cdd-83b6-2637c9202024, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980526271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Crucible stats registered with UUID: 9017d204-fc9d-4cdd-83b6-2637c9202024","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.9805361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:04.980545309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980678988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980695076Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980704865Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980868851Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.98088522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"9017d204-fc9d-4cdd-83b6-2637c9202024 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980928546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980937895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980947544Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980958323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980967543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980977112Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.980986791Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"ds_connection connected from Ok(127.0.0.1:53290)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981035367Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"ds_connection connected from Ok(127.0.0.1:50113)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981046216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"ds_connection connected from Ok(127.0.0.1:57611)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981056405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981768143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981780982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.981791631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982095444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982130051Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 38, 29, 31, 30, 38, 38, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982183157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[0]R generation[0..12]: [13, 10, 24, 19, 21, 20, 24, 24, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982194186Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982204985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 38, 29, 31, 30, 38, 38, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982217064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[1]R generation[0..12]: [13, 10, 24, 19, 21, 20, 24, 24, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982227213Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982238182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 38, 29, 31, 30, 38, 38, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982248381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[2]R generation[0..12]: [13, 10, 24, 19, 21, 20, 24, 24, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.98225861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982269479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982278738Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982288277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Next flush: 41","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982297437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982306696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"":"downstairs","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982316105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982324674Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"0","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982334143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"1","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982343573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"client":"2","":"downstairs","session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982353262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982363391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"9017d204-fc9d-4cdd-83b6-2637c9202024 is now active with session: 6c657847-5beb-4b24-9b12-16f0b1eb2925","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.98237313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982381679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"session_id":"6c657847-5beb-4b24-9b12-16f0b1eb2925"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982390619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Activated sub_volume 9017d204-fc9d-4cdd-83b6-2637c9202024","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982398598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982406607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.982699582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} 01/30 Write block 276 len 12288 data: 2 2 5 02/30 Read block 361 len 28672 03/30 Write block 247 len 20480 data: 4 5 5 5 4 04/30 Read block 85 len 16384 05/30 Write block 370 len 40960 data: 4 4 3 3 4 5 6 6 5 4 06/30 Read block 575 len 4096 07/30 Read block 196 len 12288 08/30 Write block 134 len 32768 data: 3 3 3 3 3 4 4 4 09/30 Read block 493 len 24576 10/30 Write block 489 len 20480 data: 5 5 4 4 4 11/30 Write block 450 len 24576 data: 6 4 3 4 5 5 12/30 Read block 314 len 28672 13/30 Read block 536 len 8192 14/30 Write block 318 len 16384 data: 5 5 5 5 15/30 Read block 479 len 36864 16/30 Read block 593 len 8192 17/30 Read block 506 len 36864 18/30 Write block 375 len 36864 data: 6 7 7 6 5 4 5 4 4 19/30 Read block 43 len 16384 20/30 Write block 501 len 28672 data: 2 3 3 4 4 5 6 21/30 Write block 107 len 16384 data: 2 2 2 2 22/30 Read block 343 len 24576 23/30 Write block 478 len 16384 data: 6 5 4 6 24/30 Read block 176 len 8192 25/30 Read block 563 len 8192 26/30 Write block 41 len 24576 data: 2 2 2 2 2 3 27/30 Read block 256 len 36864 28/30 Read block 19 len 24576 29/30 Write block 98 len 28672 data: 3 4 4 4 4 4 4 30/30 Write block 450 len 8192 data: 7 5 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 8 Done Done Done false 1008 Acked Read 6 Done Done Done false 1009 Acked Write 5 Done Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 2 2 24 Done 10 28 28 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:04.99201247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:04.992184115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1237,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 24 24 24 38 38 38 F T T <--- 3 060-079 19 19 19 29 29 29 F F F 4 080-099 21 21 21 31 31 31 F T T <--- 5 100-119 20 20 20 30 30 30 F T T <--- 6 120-139 24 24 24 38 38 38 T T T 7 140-159 24 24 24 38 38 38 T T T 8 160-179 25 25 25 40 40 40 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 22 22 22 35 35 35 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 25 25 25 40 40 40 T T T 13 260-279 24 24 24 39 39 39 T T T 14 280-299 25 25 25 40 40 40 F F F 15 300-319 25 25 25 40 40 40 F T T <--- 16 320-339 24 24 24 38 38 38 F T T <--- 17 340-359 24 24 24 39 39 39 F F F 18 360-379 24 24 24 39 39 39 T T T 19 380-399 25 25 25 40 40 40 F T T <--- 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 25 25 25 40 40 40 F T T <--- 23 460-479 24 24 24 39 39 39 F T T <--- 24 480-499 22 22 22 35 35 35 T T T 25 500-519 20 20 20 30 30 30 F T T <--- 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 25, Max flush: 40 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now Files /tmp/test_repair/test_repair_verify.data and /tmp/test_repair/test_repair_verify.data.last differ 378d377 < 5, 379a379,380 > 7, > 7, 383d383 < 4, 483c483,484 < 3, --- > 4, > 6, 486d486 < 5, diff found after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 10 10 10 15 15 15 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 26 26 26 41 41 41 F F F 5 100-119 26 26 26 41 41 41 F F F 6 120-139 26 26 26 41 41 41 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 25 25 25 40 40 40 F F F 9 180-199 10 10 10 16 16 16 F F F 10 200-219 22 22 22 35 35 35 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 26 26 26 41 41 41 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 26 26 26 41 41 41 F F F 17 340-359 24 24 24 39 39 39 F F F 18 360-379 26 26 26 41 41 41 F F F 19 380-399 26 26 26 41 41 41 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 26 26 26 41 41 41 F F F 23 460-479 26 26 26 41 41 41 F F F 24 480-499 26 26 26 41 41 41 F F F 25 500-519 26 26 26 41 41 41 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 26, Max flush: 41 Begin loop 14 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.266512363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.274645744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.274945298Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.274974606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.274983055Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275008933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Crucible 34bd996b-b815-41e3-afc0-1bcc7799ccaa has session id: 8d777e6d-4423-476e-a45d-dca034e664ac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275347243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Upstairs opts: Upstairs UUID: 34bd996b-b815-41e3-afc0-1bcc7799ccaa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275376361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Crucible stats registered with UUID: 34bd996b-b815-41e3-afc0-1bcc7799ccaa","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.27538928Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:09.275398579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275532367Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275548226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275558265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275690543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275707222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"34bd996b-b815-41e3-afc0-1bcc7799ccaa active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275743409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275752638Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275762207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275771566Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275781056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275790955Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275800604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:60702)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.27584762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:60870)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275858339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:55650)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.275870358Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276517931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276543359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276554848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276831414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276841573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[0]R flush_numbers[0..12]: [22, 15, 41, 29, 41, 41, 41, 41, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276896858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[0]R generation[0..12]: [13, 10, 26, 19, 26, 26, 26, 26, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276907837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276918726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[1]R flush_numbers[0..12]: [22, 15, 41, 29, 41, 41, 41, 41, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276928926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[1]R generation[0..12]: [13, 10, 26, 19, 26, 26, 26, 26, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276938975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276949594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[2]R flush_numbers[0..12]: [22, 15, 41, 29, 41, 41, 41, 41, 40, 16, 35, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276959703Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[2]R generation[0..12]: [13, 10, 26, 19, 26, 26, 26, 26, 25, 10, 22, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276969852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.276983061Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Max found gen is 27","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.2769925Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Generation requested: 28 >= found:27","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277017418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Next flush: 42","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277026617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277035866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"":"downstairs","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277045295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277053895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"0","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277063324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"1","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277072593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"client":"2","":"downstairs","session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277082132Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277090981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"34bd996b-b815-41e3-afc0-1bcc7799ccaa is now active with session: 8d777e6d-4423-476e-a45d-dca034e664ac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277100571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.27710903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"session_id":"8d777e6d-4423-476e-a45d-dca034e664ac"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277118349Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Activated sub_volume 34bd996b-b815-41e3-afc0-1bcc7799ccaa","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277126308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277134088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.277407794Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} 01/30 Write block 362 len 32768 data: 6 6 6 6 5 5 5 5 02/30 Read block 271 len 4096 03/30 Read block 249 len 4096 04/30 Read block 54 len 8192 05/30 Read block 507 len 4096 06/30 Read block 198 len 4096 07/30 Write block 273 len 28672 data: 3 3 2 3 3 6 5 08/30 Write block 212 len 32768 data: 4 3 3 3 3 4 4 5 09/30 Write block 333 len 36864 data: 2 2 3 3 2 3 3 4 3 10/30 Read block 316 len 8192 11/30 Read block 498 len 4096 12/30 Read block 158 len 24576 13/30 Read block 495 len 20480 14/30 Write block 195 len 4096 data: 2 15/30 Read block 167 len 32768 16/30 Read block 2 len 24576 17/30 Write block 24 len 16384 data: 3 3 3 3 18/30 Write block 460 len 24576 data: 4 4 5 5 4 4 19/30 Read block 339 len 36864 20/30 Read block 244 len 20480 21/30 Read block 505 len 36864 22/30 Read block 152 len 24576 23/30 Write block 392 len 12288 data: 6 3 3 24/30 Read block 273 len 4096 25/30 Flush 26/30 Write block 337 len 24576 data: 3 4 4 5 4 3 27/30 Read block 499 len 40960 28/30 Write block 277 len 4096 data: 4 29/30 Write block 168 len 24576 data: 2 2 3 3 2 3 30/30 Write block 442 len 12288 data: 6 6 6 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Write 7 Done Done Done false 1007 Acked Write 8 Done Done Done false 1008 Acked Write 9 Done Done Done false 1009 Acked Read 2 Done Done Done false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 20 24 Done 28 28 10 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:09.285445914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:09.285618209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1248,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 28 28 10 42 42 15 F F F <--- 2 040-059 26 26 26 41 41 41 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 26 26 26 41 41 41 F F F 5 100-119 26 26 26 41 41 41 F F F 6 120-139 26 26 26 41 41 41 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 25 25 25 40 40 40 T T F <--- 9 180-199 28 28 10 42 42 16 F F F <--- 10 200-219 28 28 22 42 42 35 F F T <--- 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 28 28 26 42 42 41 T T T <--- 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 28 28 26 42 42 41 T T T <--- 17 340-359 28 28 24 42 42 39 T T T <--- 18 360-379 28 28 26 42 42 41 F F T <--- 19 380-399 28 28 26 42 42 41 F F F <--- 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 26 26 26 41 41 41 T T F <--- 23 460-479 28 28 26 42 42 41 F F F <--- 24 480-499 26 26 26 41 41 41 F F F 25 500-519 26 26 26 41 41 41 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 28, Max flush: 42 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 26 26 26 41 41 41 F F F 3 060-079 19 19 19 29 29 29 F F F 4 080-099 26 26 26 41 41 41 F F F 5 100-119 26 26 26 41 41 41 F F F 6 120-139 26 26 26 41 41 41 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 29 29 29 43 43 43 F F F 9 180-199 28 28 28 42 42 42 F F F 10 200-219 28 28 28 42 42 42 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 29 29 29 43 43 43 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 29 29 29 43 43 43 F F F 17 340-359 29 29 29 43 43 43 F F F 18 360-379 28 28 28 42 42 42 F F F 19 380-399 28 28 28 42 42 42 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 29 29 29 43 43 43 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 26 26 26 41 41 41 F F F 25 500-519 26 26 26 41 41 41 F F F 26 520-539 22 22 22 34 34 34 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 29, Max flush: 43 Begin loop 15 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.532113651Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.540316457Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.540635789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.540663046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.540675155Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.540686464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Crucible 9ce1dc2f-9649-4faf-8765-1053fcbfa3e5 has session id: 75d03401-2b8f-4131-a016-2c4b586b2db7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541016835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Upstairs opts: Upstairs UUID: 9ce1dc2f-9649-4faf-8765-1053fcbfa3e5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541063661Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Crucible stats registered with UUID: 9ce1dc2f-9649-4faf-8765-1053fcbfa3e5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541073981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:13.54108305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541219718Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541249795Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541260694Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541387243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541406412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"9ce1dc2f-9649-4faf-8765-1053fcbfa3e5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541429989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541439169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541448598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541458007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541467276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541476955Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541486795Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"ds_connection connected from Ok(127.0.0.1:49727)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541565218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"ds_connection connected from Ok(127.0.0.1:57496)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.541589646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"ds_connection connected from Ok(127.0.0.1:39371)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.54165865Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542185764Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542203452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542219001Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542519545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542528944Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[0]R flush_numbers[0..12]: [22, 42, 41, 29, 41, 41, 41, 41, 43, 42, 42, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542554422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[0]R generation[0..12]: [13, 28, 26, 19, 26, 26, 26, 26, 29, 28, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542564761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.54257559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[1]R flush_numbers[0..12]: [22, 42, 41, 29, 41, 41, 41, 41, 43, 42, 42, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542585989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[1]R generation[0..12]: [13, 28, 26, 19, 26, 26, 26, 26, 29, 28, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542596248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542607047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[2]R flush_numbers[0..12]: [22, 42, 41, 29, 41, 41, 41, 41, 43, 42, 42, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542617216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[2]R generation[0..12]: [13, 28, 26, 19, 26, 26, 26, 26, 29, 28, 28, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542627285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542637964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542647333Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542656913Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Next flush: 44","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542665972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542675171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"":"downstairs","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.54268468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542694279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"0","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542703979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"1","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542713618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"client":"2","":"downstairs","session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542723437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542732456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"9ce1dc2f-9649-4faf-8765-1053fcbfa3e5 is now active with session: 75d03401-2b8f-4131-a016-2c4b586b2db7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542742185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542750724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"session_id":"75d03401-2b8f-4131-a016-2c4b586b2db7"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542759704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Activated sub_volume 9ce1dc2f-9649-4faf-8765-1053fcbfa3e5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542767543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.542780052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.543044739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} 01/30 Write block 124 len 32768 data: 4 4 4 4 4 3 3 3 02/30 Write block 444 len 36864 data: 7 4 6 6 7 6 8 6 4 03/30 Read block 327 len 40960 04/30 Write block 83 len 32768 data: 2 2 2 3 3 4 4 4 05/30 Write block 207 len 20480 data: 4 3 3 3 4 06/30 Read block 38 len 16384 07/30 Write block 534 len 4096 data: 2 08/30 Read block 519 len 28672 09/30 Flush 10/30 Write block 359 len 12288 data: 5 5 6 11/30 Read block 162 len 20480 12/30 Read block 46 len 4096 13/30 Read block 361 len 16384 14/30 Read block 423 len 32768 15/30 Read block 145 len 36864 16/30 Read block 79 len 24576 17/30 Write block 409 len 32768 data: 2 2 2 2 2 2 2 3 18/30 Read block 0 len 16384 19/30 Write block 361 len 36864 data: 7 7 7 7 7 6 6 6 6 20/30 Read block 551 len 40960 21/30 Write block 59 len 12288 data: 5 5 6 22/30 Read block 510 len 20480 23/30 Flush 24/30 Write block 127 len 24576 data: 5 5 4 4 4 3 25/30 Write block 361 len 32768 data: 8 8 8 8 8 7 7 7 26/30 Write block 215 len 4096 data: 4 27/30 Read block 267 len 12288 28/30 Read block 387 len 40960 29/30 Write block 5 len 20480 data: 3 3 3 3 3 30/30 Read block 462 len 28672 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 30 5 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:13.552046595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:13.552223009Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1256,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 22 22 22 T F T <--- 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 26 30 45 41 45 F F F <--- 3 060-079 30 19 30 45 29 45 F F F <--- 4 080-099 30 26 30 44 41 44 F T F <--- 5 100-119 26 26 26 41 41 41 F F F 6 120-139 30 26 30 44 41 44 T T T <--- 7 140-159 26 26 26 41 41 41 F F F 8 160-179 29 29 29 43 43 43 F F F 9 180-199 28 28 28 42 42 42 F F F 10 200-219 30 28 30 44 42 44 T T T <--- 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 29 29 29 43 43 43 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 29 29 29 43 43 43 F F F 17 340-359 30 29 30 45 43 45 F F F <--- 18 360-379 30 28 30 45 42 45 T F T <--- 19 380-399 28 28 28 42 42 42 F F F 20 400-419 30 22 30 45 32 45 F F F <--- 21 420-439 19 19 19 29 29 29 F F F 22 440-459 30 29 30 44 43 44 F T F <--- 23 460-479 28 28 28 42 42 42 F F F 24 480-499 26 26 26 41 41 41 F F F 25 500-519 26 26 26 41 41 41 F F F 26 520-539 30 22 30 44 34 44 F F F <--- 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 30, Max flush: 45 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 46 46 46 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F F F 3 060-079 30 30 30 45 45 45 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 26 26 26 41 41 41 F F F 6 120-139 31 31 31 46 46 46 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 29 29 29 43 43 43 F F F 9 180-199 28 28 28 42 42 42 F F F 10 200-219 31 31 31 46 46 46 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 29 29 29 43 43 43 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 F F F 16 320-339 29 29 29 43 43 43 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 31 31 31 46 46 46 F F F 19 380-399 28 28 28 42 42 42 F F F 20 400-419 30 30 30 45 45 45 F F F 21 420-439 19 19 19 29 29 29 F F F 22 440-459 30 30 30 44 44 44 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 26 26 26 41 41 41 F F F 25 500-519 26 26 26 41 41 41 F F F 26 520-539 30 30 30 44 44 44 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 31, Max flush: 46 Begin loop 16 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.798083377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806261255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80654781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806575867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806584626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806596035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Crucible 234a87d4-7b58-497b-ad01-3a960b7cdc3b has session id: f56dfdd0-b4c3-48ac-abc1-b119962d466e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806959824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Upstairs opts: Upstairs UUID: 234a87d4-7b58-497b-ad01-3a960b7cdc3b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806983882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Crucible stats registered with UUID: 234a87d4-7b58-497b-ad01-3a960b7cdc3b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.806993641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:17.80700245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807138298Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807154287Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807164256Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80734355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807362109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"234a87d4-7b58-497b-ad01-3a960b7cdc3b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807387287Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807396556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807406185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807415824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807425183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807434752Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807446201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"ds_connection connected from Ok(127.0.0.1:40935)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807483518Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"ds_connection connected from Ok(127.0.0.1:46994)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807509656Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64254)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.807611757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808124482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808144221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808177958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808419597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808454034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[0]R flush_numbers[0..12]: [46, 42, 45, 45, 44, 41, 46, 41, 43, 42, 46, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80850078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[0]R generation[0..12]: [31, 28, 30, 30, 30, 26, 31, 26, 29, 28, 31, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808511949Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808522858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[1]R flush_numbers[0..12]: [46, 42, 45, 45, 44, 41, 46, 41, 43, 42, 46, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808533117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[1]R generation[0..12]: [31, 28, 30, 30, 30, 26, 31, 26, 29, 28, 31, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808543326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808555675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[2]R flush_numbers[0..12]: [46, 42, 45, 45, 44, 41, 46, 41, 43, 42, 46, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808565804Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[2]R generation[0..12]: [31, 28, 30, 30, 30, 26, 31, 26, 29, 28, 31, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808576003Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808586662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808595771Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80860525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80861448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808623659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"":"downstairs","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808633208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808641677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"0","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808651056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"1","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808660416Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"client":"2","":"downstairs","session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808669985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808678894Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"234a87d4-7b58-497b-ad01-3a960b7cdc3b is now active with session: f56dfdd0-b4c3-48ac-abc1-b119962d466e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808688633Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808697182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"session_id":"f56dfdd0-b4c3-48ac-abc1-b119962d466e"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808707162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Activated sub_volume 234a87d4-7b58-497b-ad01-3a960b7cdc3b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.808714971Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.80872286Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.809083219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} 01/30 Write block 9 len 20480 data: 4 3 3 4 3 02/30 Flush 03/30 Write block 167 len 40960 data: 2 3 3 4 4 3 4 2 2 2 04/30 Write block 131 len 16384 data: 5 4 3 4 05/30 Read block 435 len 4096 06/30 Read block 577 len 16384 07/30 Flush 08/30 Write block 205 len 12288 data: 5 5 5 09/30 Write block 328 len 4096 data: 3 10/30 Flush 11/30 Write block 107 len 8192 data: 3 3 12/30 Read block 128 len 24576 13/30 Write block 325 len 40960 data: 3 3 3 4 3 4 3 3 3 3 14/30 Read block 117 len 40960 15/30 Write block 497 len 12288 data: 2 2 2 16/30 Flush 17/30 Flush 18/30 Write block 428 len 28672 data: 4 4 5 3 3 4 3 19/30 Write block 510 len 4096 data: 4 20/30 Write block 191 len 36864 data: 2 2 2 2 3 2 2 2 2 21/30 Read block 457 len 28672 22/30 Write block 402 len 28672 data: 8 7 7 6 4 4 2 23/30 Read block 273 len 40960 24/30 Read block 292 len 16384 25/30 Write block 306 len 4096 data: 4 26/30 Write block 367 len 12288 data: 8 8 7 27/30 Read block 376 len 4096 28/30 Read block 82 len 8192 29/30 Write block 391 len 20480 data: 6 7 4 4 3 30/30 Read block 532 len 24576 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:17.817397834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:17.817569029Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1264,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 31 32 47 46 47 F T F <--- 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F F F 3 060-079 30 30 30 45 45 45 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 32 26 32 50 41 50 F F F <--- 6 120-139 32 31 32 48 46 48 F F F <--- 7 140-159 26 26 26 41 41 41 F F F 8 160-179 32 29 32 48 43 48 F F F <--- 9 180-199 28 28 28 42 42 42 T F T <--- 10 200-219 32 31 32 49 46 49 F F F <--- 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 29 29 29 43 43 43 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 26 26 26 41 41 41 T F T <--- 16 320-339 32 29 32 50 43 50 F F F <--- 17 340-359 30 30 30 45 45 45 F F F 18 360-379 31 31 31 46 46 46 T F T <--- 19 380-399 28 28 28 42 42 42 T F T <--- 20 400-419 30 30 30 45 45 45 T F T <--- 21 420-439 19 19 19 29 29 29 T F T <--- 22 440-459 30 30 30 44 44 44 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 32 26 32 50 41 50 F F F <--- 25 500-519 26 26 26 41 41 41 T F T <--- 26 520-539 30 30 30 44 44 44 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 32, Max flush: 50 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 47 47 47 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F F F 3 060-079 30 30 30 45 45 45 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 32 32 32 50 50 50 F F F 6 120-139 32 32 32 48 48 48 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 32 32 32 48 48 48 F F F 9 180-199 33 33 33 51 51 51 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 22 22 22 36 36 36 F F F 12 240-259 26 26 26 41 41 41 F F F 13 260-279 29 29 29 43 43 43 F F F 14 280-299 25 25 25 40 40 40 F F F 15 300-319 33 33 33 51 51 51 F F F 16 320-339 32 32 32 50 50 50 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 33 33 33 51 51 51 F F F 19 380-399 33 33 33 51 51 51 F F F 20 400-419 33 33 33 51 51 51 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 30 30 30 44 44 44 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 33 33 33 51 51 51 F F F 26 520-539 30 30 30 44 44 44 F F F 27 540-559 22 22 22 32 32 32 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 22 22 22 34 34 34 F F F Max gen: 33, Max flush: 51 Begin loop 17 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.068065123Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:22.0752209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.075275395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082278745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082522404Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082545902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082554331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.08256567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Crucible ae33eddc-2a03-4762-bc98-5204e912a25e has session id: 4649a74a-c5d3-497a-84f1-6d7a6994aa83","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082896721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Upstairs opts: Upstairs UUID: ae33eddc-2a03-4762-bc98-5204e912a25e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082925749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Crucible stats registered with UUID: ae33eddc-2a03-4762-bc98-5204e912a25e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.082936808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:22.082948077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083077995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083108193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083118942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.08325406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083269919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"ae33eddc-2a03-4762-bc98-5204e912a25e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083301556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083310785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083320324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083329663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083338963Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083348612Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083358431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"ds_connection connected from Ok(127.0.0.1:39871)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083436494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"ds_connection connected from Ok(127.0.0.1:40992)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083537005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"ds_connection connected from Ok(127.0.0.1:55247)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.083561583Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084036542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084079958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.08416858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084356814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084366353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[0]R flush_numbers[0..12]: [47, 42, 45, 45, 44, 50, 48, 41, 48, 51, 49, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084470054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[0]R generation[0..12]: [32, 28, 30, 30, 30, 32, 32, 26, 32, 33, 32, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084481013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084491892Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[1]R flush_numbers[0..12]: [47, 42, 45, 45, 44, 50, 48, 41, 48, 51, 49, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084502331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[1]R generation[0..12]: [32, 28, 30, 30, 30, 32, 32, 26, 32, 33, 32, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.08451251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084523309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[2]R flush_numbers[0..12]: [47, 42, 45, 45, 44, 50, 48, 41, 48, 51, 49, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084533519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[2]R generation[0..12]: [32, 28, 30, 30, 30, 32, 32, 26, 32, 33, 32, 22]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084543688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084554557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084565526Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084574985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Next flush: 52","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084584154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084593443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"":"downstairs","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084603362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084612132Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"0","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084621951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"1","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.08463178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"client":"2","":"downstairs","session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084641509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084650408Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"ae33eddc-2a03-4762-bc98-5204e912a25e is now active with session: 4649a74a-c5d3-497a-84f1-6d7a6994aa83","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084660158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084668707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"session_id":"4649a74a-c5d3-497a-84f1-6d7a6994aa83"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084677526Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Activated sub_volume ae33eddc-2a03-4762-bc98-5204e912a25e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084685285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.084693205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.085016546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} 01/30 Write block 110 len 16384 data: 3 2 2 3 02/30 Read block 287 len 8192 03/30 Write block 316 len 8192 data: 4 5 04/30 Write block 115 len 40960 data: 3 3 3 4 4 4 4 5 4 5 05/30 Write block 589 len 40960 data: 7 7 7 6 4 4 4 4 3 2 06/30 Write block 302 len 36864 data: 6 7 6 5 5 4 4 4 4 07/30 Flush 08/30 Write block 512 len 24576 data: 5 5 5 5 4 4 09/30 Read block 37 len 28672 10/30 Read block 579 len 36864 11/30 Read block 303 len 40960 12/30 Flush 13/30 Write block 535 len 24576 data: 3 4 3 3 3 2 14/30 Write block 238 len 32768 data: 3 3 3 3 2 2 3 3 15/30 Write block 172 len 12288 data: 4 5 3 16/30 Write block 272 len 36864 data: 4 4 4 3 4 5 7 6 5 17/30 Read block 407 len 36864 18/30 Write block 534 len 4096 data: 3 19/30 Flush 20/30 Flush 21/30 Write block 533 len 8192 data: 2 4 22/30 Write block 69 len 20480 data: 2 2 2 2 2 23/30 Read block 526 len 40960 24/30 Write block 235 len 16384 data: 6 5 5 4 25/30 Read block 318 len 40960 26/30 Flush 27/30 Write block 237 len 24576 data: 6 5 4 4 4 3 28/30 Write block 584 len 4096 data: 6 29/30 Write block 253 len 40960 data: 4 3 3 3 3 4 3 3 6 5 30/30 Read block 297 len 4096 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:18 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1012 Acked Write 6 Done Done Done false 1013 Acked Write 8 Done Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 10 Sent Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 16 0 2 18 Done 2 18 16 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1011 1010 1009 1008 1007 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:22.093613947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:22.093767854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1272,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 47 47 47 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F F F 3 060-079 30 34 34 45 56 56 F F F <--- 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 34 34 34 52 52 52 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 32 34 34 48 54 54 F F F <--- 9 180-199 33 33 33 51 51 51 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 22 34 34 36 56 56 T T T <--- 12 240-259 26 34 34 41 54 54 T T T <--- 13 260-279 29 34 34 43 54 54 F T T <--- 14 280-299 25 34 34 40 54 54 F F F <--- 15 300-319 34 34 34 52 52 52 F F F 16 320-339 32 32 32 50 50 50 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 33 33 33 51 51 51 F F F 19 380-399 33 33 33 51 51 51 F F F 20 400-419 33 33 33 51 51 51 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 30 30 30 44 44 44 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 34 34 34 53 53 53 F F F 26 520-539 30 34 34 44 56 56 T F F <--- 27 540-559 22 34 34 32 54 54 T F F <--- 28 560-579 23 23 23 37 37 37 F F F 29 580-599 34 34 34 52 52 52 F T T <--- Max gen: 34, Max flush: 56 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 47 47 47 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F F F 3 060-079 34 34 34 56 56 56 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 34 34 34 52 52 52 F F F 7 140-159 26 26 26 41 41 41 F F F 8 160-179 34 34 34 54 54 54 F F F 9 180-199 33 33 33 51 51 51 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 35 35 35 57 57 57 F F F 12 240-259 35 35 35 57 57 57 F F F 13 260-279 35 35 35 57 57 57 F F F 14 280-299 34 34 34 54 54 54 F F F 15 300-319 34 34 34 52 52 52 F F F 16 320-339 32 32 32 50 50 50 F F F 17 340-359 30 30 30 45 45 45 F F F 18 360-379 33 33 33 51 51 51 F F F 19 380-399 33 33 33 51 51 51 F F F 20 400-419 33 33 33 51 51 51 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 30 30 30 44 44 44 F F F 23 460-479 28 28 28 42 42 42 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 34 34 34 53 53 53 F F F 26 520-539 34 34 34 56 56 56 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 35 35 35 57 57 57 F F F Max gen: 35, Max flush: 57 Begin loop 18 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.340897041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:26.348107673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.348163158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.354961756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355239492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35526287Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355271139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355282618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Crucible 5f53de55-46da-4f8c-8397-5c81324f56c7 has session id: f2d3219c-161f-44b2-a70c-0312aaea3a9b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355630168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Upstairs opts: Upstairs UUID: 5f53de55-46da-4f8c-8397-5c81324f56c7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355651766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Crucible stats registered with UUID: 5f53de55-46da-4f8c-8397-5c81324f56c7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355661675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:26.355670494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355809572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355826891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35583688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355976658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.355991906Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"5f53de55-46da-4f8c-8397-5c81324f56c7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356008875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356017644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356028733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356038212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356047442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356057201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35606678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37918)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356110006Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"ds_connection connected from Ok(127.0.0.1:45121)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35618048Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"ds_connection connected from Ok(127.0.0.1:58322)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356204538Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356729882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356761839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.356861151Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357060653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357070872Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[0]R flush_numbers[0..12]: [47, 42, 45, 56, 44, 52, 52, 41, 54, 51, 49, 57]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357140916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[0]R generation[0..12]: [32, 28, 30, 34, 30, 34, 34, 26, 34, 33, 32, 35]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357151765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357165094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[1]R flush_numbers[0..12]: [47, 42, 45, 56, 44, 52, 52, 41, 54, 51, 49, 57]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357175243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[1]R generation[0..12]: [32, 28, 30, 34, 30, 34, 34, 26, 34, 33, 32, 35]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357185352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357195931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[2]R flush_numbers[0..12]: [47, 42, 45, 56, 44, 52, 52, 41, 54, 51, 49, 57]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357206111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[2]R generation[0..12]: [32, 28, 30, 34, 30, 34, 34, 26, 34, 33, 32, 35]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35721626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357226999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357236158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357245687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Next flush: 58","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357254666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357263926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"":"downstairs","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357273345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357281804Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"0","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357291133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"1","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357300452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"client":"2","":"downstairs","session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357311391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357320351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"5f53de55-46da-4f8c-8397-5c81324f56c7 is now active with session: f2d3219c-161f-44b2-a70c-0312aaea3a9b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.35732997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357338399Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"session_id":"f2d3219c-161f-44b2-a70c-0312aaea3a9b"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357347298Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Activated sub_volume 5f53de55-46da-4f8c-8397-5c81324f56c7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357355048Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357362917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.357700598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} 01/30 Write block 349 len 32768 data: 4 5 4 4 4 5 4 4 02/30 Read block 353 len 20480 03/30 Read block 345 len 36864 04/30 Read block 409 len 12288 05/30 Write block 278 len 16384 data: 8 7 6 5 06/30 Read block 413 len 16384 07/30 Read block 50 len 16384 08/30 Write block 584 len 40960 data: 7 6 8 6 6 8 8 8 7 5 09/30 Write block 226 len 28672 data: 2 3 3 3 3 3 4 10/30 Read block 62 len 16384 11/30 Write block 311 len 16384 data: 3 2 2 3 12/30 Write block 153 len 40960 data: 5 5 5 4 5 5 4 2 2 2 13/30 Write block 522 len 12288 data: 5 3 5 14/30 Read block 527 len 20480 15/30 Write block 441 len 8192 data: 5 7 16/30 Write block 193 len 24576 data: 3 3 4 3 3 3 17/30 Write block 509 len 32768 data: 5 5 4 6 6 6 6 5 18/30 Flush 19/30 Write block 474 len 20480 data: 7 7 7 6 7 20/30 Read block 156 len 16384 21/30 Read block 65 len 24576 22/30 Flush 23/30 Write block 52 len 24576 data: 5 5 6 4 3 3 24/30 Read block 159 len 12288 25/30 Read block 537 len 32768 26/30 Read block 252 len 20480 27/30 Read block 469 len 36864 28/30 Read block 381 len 12288 29/30 Write block 188 len 20480 data: 3 2 2 3 3 30/30 Read block 302 len 16384 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:26.366641339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:26.366833372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1280,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 47 47 47 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 30 30 30 45 45 45 F T T <--- 3 060-079 34 34 34 56 56 56 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 34 34 34 52 52 52 F F F 7 140-159 26 36 36 41 58 58 F F F <--- 8 160-179 34 36 36 54 58 58 F F F <--- 9 180-199 33 36 36 51 58 58 F T T <--- 10 200-219 32 32 32 49 49 49 F F F 11 220-239 35 36 36 57 58 58 F F F <--- 12 240-259 35 35 35 57 57 57 F F F 13 260-279 35 36 36 57 58 58 F F F <--- 14 280-299 34 36 36 54 58 58 F F F <--- 15 300-319 34 36 36 52 58 58 F F F <--- 16 320-339 32 32 32 50 50 50 F F F 17 340-359 30 36 36 45 58 58 T F F <--- 18 360-379 33 33 33 51 51 51 F F F 19 380-399 33 33 33 51 51 51 F F F 20 400-419 33 33 33 51 51 51 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 30 36 36 44 58 58 F F F <--- 23 460-479 28 36 36 42 59 59 F F F <--- 24 480-499 32 32 32 50 50 50 F F F 25 500-519 34 36 36 53 58 58 F F F <--- 26 520-539 34 36 36 56 58 58 F F F <--- 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 35 36 36 57 58 58 F F F <--- Max gen: 36, Max flush: 59 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 47 47 47 F F F 1 020-039 28 28 28 42 42 42 F F F 2 040-059 37 37 37 60 60 60 F F F 3 060-079 34 34 34 56 56 56 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 34 34 34 52 52 52 F F F 7 140-159 36 36 36 58 58 58 F F F 8 160-179 36 36 36 58 58 58 F F F 9 180-199 37 37 37 60 60 60 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 36 36 36 58 58 58 F F F 12 240-259 35 35 35 57 57 57 F F F 13 260-279 36 36 36 58 58 58 F F F 14 280-299 36 36 36 58 58 58 F F F 15 300-319 36 36 36 58 58 58 F F F 16 320-339 32 32 32 50 50 50 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 33 33 33 51 51 51 F F F 19 380-399 33 33 33 51 51 51 F F F 20 400-419 33 33 33 51 51 51 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 36 36 36 58 58 58 F F F 23 460-479 36 36 36 59 59 59 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 36 36 36 58 58 58 F F F 26 520-539 36 36 36 58 58 58 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 36 36 36 58 58 58 F F F Max gen: 37, Max flush: 60 Begin loop 19 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.621505802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.629835756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630112292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.63013607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630144849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630156288Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Crucible 57ee454b-36b6-412d-b59e-3b83439604ed has session id: d7ad5c72-f431-442b-88b4-0deea65690fb","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630460722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Upstairs opts: Upstairs UUID: 57ee454b-36b6-412d-b59e-3b83439604ed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.63048128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Crucible stats registered with UUID: 57ee454b-36b6-412d-b59e-3b83439604ed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630493139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:30.630501988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630636897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630652985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630662874Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630818521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630833959Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"57ee454b-36b6-412d-b59e-3b83439604ed active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630846848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630855467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630864897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630874396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630883765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630893674Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630903343Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:63402)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630985406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:41984)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.630997645Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:63665)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.631117345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.631708863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.631771718Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.631834622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632033255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632047154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[0]R flush_numbers[0..12]: [47, 42, 60, 56, 44, 52, 52, 58, 58, 60, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632130466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[0]R generation[0..12]: [32, 28, 37, 34, 30, 34, 34, 36, 36, 37, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632141435Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632152304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[1]R flush_numbers[0..12]: [47, 42, 60, 56, 44, 52, 52, 58, 58, 60, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632162534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[1]R generation[0..12]: [32, 28, 37, 34, 30, 34, 34, 36, 36, 37, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632172683Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632183462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[2]R flush_numbers[0..12]: [47, 42, 60, 56, 44, 52, 52, 58, 58, 60, 49, 58]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632193631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[2]R generation[0..12]: [32, 28, 37, 34, 30, 34, 34, 36, 36, 37, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.63220604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632217109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632226378Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632235847Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Next flush: 61","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632245086Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632254296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"":"downstairs","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632263865Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632272434Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"0","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632281873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"1","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632291572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"client":"2","":"downstairs","session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632301431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632310301Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"57ee454b-36b6-412d-b59e-3b83439604ed is now active with session: d7ad5c72-f431-442b-88b4-0deea65690fb","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-09-18T03:29:30.63232013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632333229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"session_id":"d7ad5c72-f431-442b-88b4-0deea65690fb"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632342078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Activated sub_volume 57ee454b-36b6-412d-b59e-3b83439604ed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632349937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632358886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.632592356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} 01/30 Write block 19 len 8192 data: 3 2 02/30 Flush 03/30 Write block 226 len 20480 data: 3 4 4 4 4 04/30 Read block 335 len 4096 05/30 Read block 383 len 32768 06/30 Write block 392 len 20480 data: 8 5 5 4 3 07/30 Write block 130 len 40960 data: 5 6 5 4 5 4 4 4 4 5 08/30 Flush 09/30 Write block 369 len 40960 data: 8 5 5 4 4 5 6 7 7 6 10/30 Read block 567 len 24576 11/30 Read block 354 len 36864 12/30 Write block 365 len 4096 data: 9 13/30 Read block 389 len 40960 14/30 Write block 323 len 28672 data: 6 6 4 4 4 5 4 15/30 Write block 463 len 28672 data: 6 5 5 4 4 4 5 16/30 Read block 95 len 16384 17/30 Write block 295 len 36864 data: 6 6 6 6 8 7 6 7 8 18/30 Write block 319 len 24576 data: 6 6 6 7 7 7 19/30 Write block 150 len 20480 data: 3 4 4 6 6 20/30 Read block 563 len 24576 21/30 Write block 67 len 40960 data: 3 3 3 3 3 3 3 2 2 3 22/30 Write block 31 len 24576 data: 4 4 4 4 4 4 23/30 Write block 373 len 40960 data: 5 6 7 8 8 7 5 5 6 5 24/30 Write block 244 len 24576 data: 4 4 4 5 6 6 25/30 Read block 407 len 28672 26/30 Read block 92 len 16384 27/30 Read block 71 len 40960 28/30 Write block 397 len 20480 data: 4 8 6 7 8 29/30 Read block 64 len 4096 30/30 Read block 359 len 36864 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 5 Done Sent Done false 1003 Acked Read 1 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) None Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:30.641809043Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:30.641981898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1288,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 32 38 61 47 61 F F F <--- 1 020-039 38 28 38 61 42 61 T F T <--- 2 040-059 37 37 37 60 60 60 F F F 3 060-079 34 34 34 56 56 56 T F T <--- 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 38 34 38 62 52 62 F F F <--- 7 140-159 36 36 36 58 58 58 T F T <--- 8 160-179 36 36 36 58 58 58 F F F 9 180-199 37 37 37 60 60 60 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 38 36 38 62 58 62 F F F <--- 12 240-259 35 35 35 57 57 57 T F T <--- 13 260-279 36 36 36 58 58 58 F F F 14 280-299 36 36 36 58 58 58 T F T <--- 15 300-319 36 36 36 58 58 58 T F T <--- 16 320-339 32 32 32 50 50 50 T F T <--- 17 340-359 36 36 36 58 58 58 F F F 18 360-379 33 33 33 51 51 51 T F T <--- 19 380-399 38 33 38 62 51 62 T F T <--- 20 400-419 33 33 33 51 51 51 T F T <--- 21 420-439 33 33 33 51 51 51 F F F 22 440-459 36 36 36 58 58 58 F F F 23 460-479 36 36 36 59 59 59 T F T <--- 24 480-499 32 32 32 50 50 50 F F F 25 500-519 36 36 36 58 58 58 F F F 26 520-539 36 36 36 58 58 58 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 36 36 36 58 58 58 F F F Max gen: 38, Max flush: 62 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 61 61 61 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 37 37 37 60 60 60 F F F 3 060-079 39 39 39 63 63 63 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 34 34 34 52 52 52 F F F 6 120-139 38 38 38 62 62 62 F F F 7 140-159 39 39 39 63 63 63 F F F 8 160-179 36 36 36 58 58 58 F F F 9 180-199 37 37 37 60 60 60 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 38 38 38 62 62 62 F F F 12 240-259 39 39 39 63 63 63 F F F 13 260-279 36 36 36 58 58 58 F F F 14 280-299 39 39 39 63 63 63 F F F 15 300-319 39 39 39 63 63 63 F F F 16 320-339 39 39 39 63 63 63 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 39 39 39 63 63 63 F F F 19 380-399 39 39 39 63 63 63 F F F 20 400-419 39 39 39 63 63 63 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 36 36 36 58 58 58 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 36 36 36 58 58 58 F F F 26 520-539 36 36 36 58 58 58 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 36 36 36 58 58 58 F F F Max gen: 39, Max flush: 63 Begin loop 20 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.89651025Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.904664109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.904952144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.904975682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.904984031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90499508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Crucible 7ede2841-3bcb-4630-b400-10aa874533d3 has session id: 61797b29-1914-479f-8b5a-e7f4cab9eb4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905367138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Upstairs opts: Upstairs UUID: 7ede2841-3bcb-4630-b400-10aa874533d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905421453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Crucible stats registered with UUID: 7ede2841-3bcb-4630-b400-10aa874533d3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905432862Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:34.905442172Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905583239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905613307Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905624056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905739826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905754624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"7ede2841-3bcb-4630-b400-10aa874533d3 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905772013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905780792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905790311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90579983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90580942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905819169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905828848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41353)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.905947497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45476)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906095135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:64306)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906119473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90660609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906626778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906688383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906911603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.906934102Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[0]R flush_numbers[0..12]: [61, 63, 60, 63, 44, 52, 62, 63, 58, 60, 49, 62]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907015044Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[0]R generation[0..12]: [38, 39, 37, 39, 30, 34, 38, 39, 36, 37, 32, 38]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907026553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907037413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[1]R flush_numbers[0..12]: [61, 63, 60, 63, 44, 52, 62, 63, 58, 60, 49, 62]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907047722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[1]R generation[0..12]: [38, 39, 37, 39, 30, 34, 38, 39, 36, 37, 32, 38]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907058121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90706888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[2]R flush_numbers[0..12]: [61, 63, 60, 63, 44, 52, 62, 63, 58, 60, 49, 62]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907079519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[2]R generation[0..12]: [38, 39, 37, 39, 30, 34, 38, 39, 36, 37, 32, 38]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907089628Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907100557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907109736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907119245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Next flush: 64","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907128305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907137474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"":"downstairs","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907149063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907157662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"0","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907167121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"1","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.90717645Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"client":"2","":"downstairs","session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.9071862Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907195479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"7ede2841-3bcb-4630-b400-10aa874533d3 is now active with session: 61797b29-1914-479f-8b5a-e7f4cab9eb4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907205228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907213757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"session_id":"61797b29-1914-479f-8b5a-e7f4cab9eb4a"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907222646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Activated sub_volume 7ede2841-3bcb-4630-b400-10aa874533d3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907230456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907238355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.907558817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} 01/30 Flush 02/30 Flush 03/30 Write block 8 len 40960 data: 4 5 4 4 5 4 3 3 3 3 04/30 Read block 415 len 24576 05/30 Write block 386 len 12288 data: 5 5 6 06/30 Write block 375 len 40960 data: 8 9 9 8 6 6 7 6 5 4 07/30 Write block 273 len 12288 data: 5 5 4 08/30 Write block 223 len 36864 data: 2 2 2 4 5 5 5 5 4 09/30 Write block 371 len 28672 data: 6 5 6 7 9 10 10 10/30 Read block 36 len 28672 11/30 Write block 116 len 4096 data: 4 12/30 Read block 42 len 40960 13/30 Read block 529 len 28672 14/30 Write block 180 len 32768 data: 3 3 3 4 4 3 3 3 15/30 Read block 282 len 12288 16/30 Read block 81 len 24576 17/30 Read block 19 len 36864 18/30 Read block 376 len 40960 19/30 Flush 20/30 Write block 103 len 32768 data: 5 5 4 3 4 4 3 4 21/30 Write block 456 len 12288 data: 4 3 3 22/30 Read block 24 len 36864 23/30 Read block 90 len 32768 24/30 Read block 592 len 12288 25/30 Write block 195 len 16384 data: 5 4 4 4 26/30 Read block 438 len 32768 27/30 Read block 201 len 36864 28/30 Read block 129 len 16384 29/30 Write block 524 len 40960 data: 6 4 4 4 3 3 2 2 2 3 30/30 Write block 413 len 24576 data: 3 3 3 4 3 3 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 10 Done Done Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 27 2 31 Done 26 1 26 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1001)) Some(JobId(1018)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:34.916380819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:34.916567602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1296,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 38 40 66 61 66 F T F <--- 1 020-039 39 39 39 63 63 63 F F F 2 040-059 37 37 37 60 60 60 F F F 3 060-079 39 39 39 63 63 63 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 40 34 40 66 52 66 T F T <--- 6 120-139 38 38 38 62 62 62 F F F 7 140-159 39 39 39 63 63 63 F F F 8 160-179 36 36 36 58 58 58 F F F 9 180-199 40 37 40 66 60 66 T F T <--- 10 200-219 32 32 32 49 49 49 F F F 11 220-239 40 38 40 66 62 66 F F F <--- 12 240-259 39 39 39 63 63 63 F F F 13 260-279 40 36 40 66 58 66 F F F <--- 14 280-299 39 39 39 63 63 63 F F F 15 300-319 39 39 39 63 63 63 F F F 16 320-339 39 39 39 63 63 63 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 40 39 40 66 63 66 F F F <--- 19 380-399 40 39 40 66 63 66 F F F <--- 20 400-419 39 39 39 63 63 63 T F T <--- 21 420-439 33 33 33 51 51 51 F F F 22 440-459 36 36 36 58 58 58 T F T <--- 23 460-479 39 39 39 63 63 63 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 36 36 36 58 58 58 F F F 26 520-539 36 36 36 58 58 58 T F T <--- 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 36 36 36 58 58 58 F F F Max gen: 40, Max flush: 66 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 40 40 40 66 66 66 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 37 37 37 60 60 60 F F F 3 060-079 39 39 39 63 63 63 F F F 4 080-099 30 30 30 44 44 44 F F F 5 100-119 41 41 41 67 67 67 F F F 6 120-139 38 38 38 62 62 62 F F F 7 140-159 39 39 39 63 63 63 F F F 8 160-179 36 36 36 58 58 58 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 32 32 32 49 49 49 F F F 11 220-239 40 40 40 66 66 66 F F F 12 240-259 39 39 39 63 63 63 F F F 13 260-279 40 40 40 66 66 66 F F F 14 280-299 39 39 39 63 63 63 F F F 15 300-319 39 39 39 63 63 63 F F F 16 320-339 39 39 39 63 63 63 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 40 40 40 66 66 66 F F F 19 380-399 40 40 40 66 66 66 F F F 20 400-419 41 41 41 67 67 67 F F F 21 420-439 33 33 33 51 51 51 F F F 22 440-459 41 41 41 67 67 67 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 32 32 32 50 50 50 F F F 25 500-519 36 36 36 58 58 58 F F F 26 520-539 41 41 41 67 67 67 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 23 23 23 37 37 37 F F F 29 580-599 36 36 36 58 58 58 F F F Max gen: 41, Max flush: 67 Begin loop 21 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.167123341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.175321327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.175601322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17563092Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.175639669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.175653358Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Crucible 58d7a1ea-cb3a-4d86-a8ea-54d245b027a1 has session id: 733a7aca-cc15-40cb-afb4-3eb6ea1577b6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176020066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Upstairs opts: Upstairs UUID: 58d7a1ea-cb3a-4d86-a8ea-54d245b027a1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176047994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Crucible stats registered with UUID: 58d7a1ea-cb3a-4d86-a8ea-54d245b027a1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176058713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:39.176067642Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17620836Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176243807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176254956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176393783Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176406662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"58d7a1ea-cb3a-4d86-a8ea-54d245b027a1 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17643828Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176447539Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176457198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176466727Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176475976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176487395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176497104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:61043)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176539491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:57820)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17655015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36721)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.176560239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17723369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177263578Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177275237Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177564781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177600678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[0]R flush_numbers[0..12]: [66, 63, 60, 63, 44, 67, 62, 63, 58, 67, 49, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177661403Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[0]R generation[0..12]: [40, 39, 37, 39, 30, 41, 38, 39, 36, 41, 32, 40]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177672152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177682911Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[1]R flush_numbers[0..12]: [66, 63, 60, 63, 44, 67, 62, 63, 58, 67, 49, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17769337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[1]R generation[0..12]: [40, 39, 37, 39, 30, 41, 38, 39, 36, 41, 32, 40]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177706189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177716978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[2]R flush_numbers[0..12]: [66, 63, 60, 63, 44, 67, 62, 63, 58, 67, 49, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177727227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[2]R generation[0..12]: [40, 39, 37, 39, 30, 41, 38, 39, 36, 41, 32, 40]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177737416Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177748065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177757245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177766584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Next flush: 68","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177775643Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177784882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"":"downstairs","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177794761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177803381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"0","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.17781295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"1","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177822429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"client":"2","":"downstairs","session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177832068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177841157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"58d7a1ea-cb3a-4d86-a8ea-54d245b027a1 is now active with session: 733a7aca-cc15-40cb-afb4-3eb6ea1577b6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177850956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177860666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"session_id":"733a7aca-cc15-40cb-afb4-3eb6ea1577b6"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177869765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Activated sub_volume 58d7a1ea-cb3a-4d86-a8ea-54d245b027a1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177877584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.177885563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.178206815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} 01/30 Write block 510 len 40960 data: 6 5 7 7 7 7 6 5 4 5 02/30 Write block 453 len 4096 data: 5 03/30 Write block 307 len 36864 data: 5 5 5 5 4 3 3 4 3 04/30 Write block 372 len 4096 data: 6 05/30 Read block 223 len 12288 06/30 Write block 319 len 36864 data: 7 7 7 8 8 8 5 5 5 07/30 Read block 570 len 16384 08/30 Read block 193 len 20480 09/30 Write block 569 len 32768 data: 4 2 2 2 2 2 2 2 10/30 Write block 562 len 16384 data: 5 4 4 4 11/30 Write block 228 len 20480 data: 6 6 6 5 5 12/30 Write block 140 len 4096 data: 5 13/30 Write block 208 len 20480 data: 4 4 4 5 5 14/30 Write block 582 len 32768 data: 3 4 8 7 9 7 7 9 15/30 Read block 392 len 16384 16/30 Write block 2 len 8192 data: 2 2 17/30 Read block 141 len 40960 18/30 Write block 383 len 28672 data: 6 5 5 6 6 7 6 19/30 Read block 74 len 36864 20/30 Read block 309 len 4096 21/30 Flush 22/30 Write block 485 len 24576 data: 4 6 6 6 6 6 23/30 Flush 24/30 Write block 521 len 32768 data: 5 6 4 7 5 5 5 4 25/30 Write block 44 len 28672 data: 3 3 4 4 6 5 4 26/30 Write block 293 len 28672 data: 4 5 7 7 7 7 9 27/30 Write block 72 len 40960 data: 4 4 3 3 4 3 3 2 3 3 28/30 Write block 428 len 20480 data: 5 5 6 4 4 29/30 Read block 366 len 40960 30/30 Read block 571 len 12288 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Write 1 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:39.186007866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:39.186181081Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1304,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 40 68 68 66 F F F <--- 1 020-039 39 39 39 63 63 63 F F F 2 040-059 37 37 37 60 60 60 T T F <--- 3 060-079 39 39 39 63 63 63 T T F <--- 4 080-099 30 30 30 44 44 44 T T F <--- 5 100-119 41 41 41 67 67 67 F F F 6 120-139 38 38 38 62 62 62 F F F 7 140-159 42 42 39 68 68 63 F F F <--- 8 160-179 36 36 36 58 58 58 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 42 42 32 68 68 49 F F F <--- 11 220-239 42 42 40 68 68 66 F F F <--- 12 240-259 39 39 39 63 63 63 F F F 13 260-279 40 40 40 66 66 66 F F F 14 280-299 39 39 39 63 63 63 T T F <--- 15 300-319 42 42 39 68 68 63 F F F <--- 16 320-339 42 42 39 68 68 63 F F F <--- 17 340-359 36 36 36 58 58 58 F F F 18 360-379 42 42 40 68 68 66 F F F <--- 19 380-399 42 42 40 68 68 66 F F F <--- 20 400-419 41 41 41 67 67 67 F F F 21 420-439 33 33 33 51 51 51 T T F <--- 22 440-459 42 42 41 68 68 67 F F F <--- 23 460-479 39 39 39 63 63 63 F F F 24 480-499 42 42 32 69 69 50 F F F <--- 25 500-519 42 42 36 68 68 58 F F F <--- 26 520-539 41 41 41 67 67 67 T T F <--- 27 540-559 34 34 34 54 54 54 F F F 28 560-579 42 42 23 68 68 37 F F F <--- 29 580-599 42 42 36 68 68 58 F F F <--- Max gen: 42, Max flush: 69 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 43 43 43 70 70 70 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 43 43 43 70 70 70 F F F 5 100-119 41 41 41 67 67 67 F F F 6 120-139 38 38 38 62 62 62 F F F 7 140-159 42 42 42 68 68 68 F F F 8 160-179 36 36 36 58 58 58 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 42 42 42 68 68 68 F F F 11 220-239 42 42 42 68 68 68 F F F 12 240-259 39 39 39 63 63 63 F F F 13 260-279 40 40 40 66 66 66 F F F 14 280-299 43 43 43 70 70 70 F F F 15 300-319 42 42 42 68 68 68 F F F 16 320-339 42 42 42 68 68 68 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 42 42 42 68 68 68 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 41 41 41 67 67 67 F F F 21 420-439 43 43 43 70 70 70 F F F 22 440-459 42 42 42 68 68 68 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 42 42 42 69 69 69 F F F 25 500-519 42 42 42 68 68 68 F F F 26 520-539 43 43 43 70 70 70 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 42 42 42 68 68 68 F F F 29 580-599 42 42 42 68 68 68 F F F Max gen: 43, Max flush: 70 Begin loop 22 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.448577748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:43.455755353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.455810368Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.462933007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463185995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463211143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463219622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463230981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Crucible 44299d1a-c139-45e4-a60f-c607a6124f7f has session id: 1e7929cc-c66c-449b-9436-6a8428f093b6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463563592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Upstairs opts: Upstairs UUID: 44299d1a-c139-45e4-a60f-c607a6124f7f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.46359215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Crucible stats registered with UUID: 44299d1a-c139-45e4-a60f-c607a6124f7f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463605189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:43.463614198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463736917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463767015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463777784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463921481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.46393658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"44299d1a-c139-45e4-a60f-c607a6124f7f active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463952549Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463961288Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463970897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463980366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463989835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.463999694Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464009354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:46317)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464065339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:45856)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464075968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:42779)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464123774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464701743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464726221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464836811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.464984949Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465008207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[0]R flush_numbers[0..12]: [68, 63, 70, 70, 70, 67, 62, 68, 58, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465120967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[0]R generation[0..12]: [42, 39, 43, 43, 43, 41, 38, 42, 36, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465141425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465152474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[1]R flush_numbers[0..12]: [68, 63, 70, 70, 70, 67, 62, 68, 58, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465162903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[1]R generation[0..12]: [42, 39, 43, 43, 43, 41, 38, 42, 36, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465172902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465183691Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[2]R flush_numbers[0..12]: [68, 63, 70, 70, 70, 67, 62, 68, 58, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.46519377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[2]R generation[0..12]: [42, 39, 43, 43, 43, 41, 38, 42, 36, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465203979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465216638Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465225958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465235327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Next flush: 71","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465244596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465253845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"":"downstairs","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465263584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465272274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"0","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465282033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"1","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465291682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"client":"2","":"downstairs","session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465301581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.46531046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"44299d1a-c139-45e4-a60f-c607a6124f7f is now active with session: 1e7929cc-c66c-449b-9436-6a8428f093b6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465320109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465328679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"session_id":"1e7929cc-c66c-449b-9436-6a8428f093b6"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465337558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Activated sub_volume 44299d1a-c139-45e4-a60f-c607a6124f7f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465345267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pidDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":1312} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465358036Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.465626693Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} 01/30 Flush 02/30 Write block 109 len 8192 data: 4 5 03/30 Read block 453 len 32768 04/30 Write block 309 len 28672 data: 6 6 5 4 4 5 4 05/30 Write block 126 len 20480 data: 5 6 6 5 6 06/30 Write block 593 len 12288 data: 6 5 5 07/30 Read block 386 len 28672 08/30 Read block 248 len 20480 09/30 Read block 111 len 24576 10/30 Write block 361 len 4096 data: 9 11/30 Flush 12/30 Write block 445 len 28672 data: 5 7 7 8 7 9 7 13/30 Read block 588 len 16384 14/30 Flush 15/30 Write block 444 len 12288 data: 8 6 8 16/30 Read block 565 len 24576 17/30 Flush 18/30 Write block 331 len 4096 data: 4 19/30 Flush 20/30 Write block 44 len 24576 data: 4 4 5 5 7 6 21/30 Read block 132 len 36864 22/30 Read block 97 len 12288 23/30 Write block 265 len 16384 data: 4 5 5 4 24/30 Write block 112 len 36864 data: 3 4 3 4 5 4 5 5 5 25/30 Write block 254 len 32768 data: 4 4 4 4 5 4 4 7 26/30 Read block 487 len 24576 27/30 Read block 12 len 4096 28/30 Write block 176 len 8192 data: 3 3 29/30 Read block 427 len 40960 30/30 Read block 555 len 8192 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 1 29 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:43.474139391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:43.474258431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1312,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 43 43 43 70 70 70 F T T <--- 3 060-079 43 43 43 70 70 70 F F F 4 080-099 43 43 43 70 70 70 F F F 5 100-119 41 44 44 67 72 72 T T T <--- 6 120-139 38 44 44 62 72 72 F T T <--- 7 140-159 42 42 42 68 68 68 F F F 8 160-179 36 36 36 58 58 58 F T T <--- 9 180-199 41 41 41 67 67 67 F F F 10 200-219 42 42 42 68 68 68 F F F 11 220-239 42 42 42 68 68 68 F F F 12 240-259 39 39 39 63 63 63 F T T <--- 13 260-279 40 40 40 66 66 66 F T T <--- 14 280-299 43 43 43 70 70 70 F F F 15 300-319 42 44 44 68 72 72 F F F <--- 16 320-339 42 44 44 68 75 75 F F F <--- 17 340-359 36 36 36 58 58 58 F F F 18 360-379 42 44 44 68 72 72 F F F <--- 19 380-399 42 42 42 68 68 68 F F F 20 400-419 41 41 41 67 67 67 F F F 21 420-439 43 43 43 70 70 70 F F F 22 440-459 42 44 44 68 74 74 F F F <--- 23 460-479 39 39 39 63 63 63 F F F 24 480-499 42 42 42 69 69 69 F F F 25 500-519 42 42 42 68 68 68 F F F 26 520-539 43 43 43 70 70 70 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 42 42 42 68 68 68 F F F 29 580-599 42 44 44 68 72 72 F F F <--- Max gen: 44, Max flush: 75 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 45 45 45 76 76 76 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 43 43 43 70 70 70 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 45 45 45 76 76 76 F F F 7 140-159 42 42 42 68 68 68 F F F 8 160-179 45 45 45 76 76 76 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 42 42 42 68 68 68 F F F 11 220-239 42 42 42 68 68 68 F F F 12 240-259 45 45 45 76 76 76 F F F 13 260-279 45 45 45 76 76 76 F F F 14 280-299 43 43 43 70 70 70 F F F 15 300-319 44 44 44 72 72 72 F F F 16 320-339 44 44 44 75 75 75 F F F 17 340-359 36 36 36 58 58 58 F F F 18 360-379 44 44 44 72 72 72 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 41 41 41 67 67 67 F F F 21 420-439 43 43 43 70 70 70 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 42 42 42 69 69 69 F F F 25 500-519 42 42 42 68 68 68 F F F 26 520-539 43 43 43 70 70 70 F F F 27 540-559 34 34 34 54 54 54 F F F 28 560-579 42 42 42 68 68 68 F F F 29 580-599 44 44 44 72 72 72 F F F Max gen: 45, Max flush: 76 Begin loop 23 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.726164492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73445088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.734725386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.734750964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.734759413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.734770602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Crucible 4e99ac97-e5aa-406b-b83e-1cceb43b3c5e has session id: ef2f87ce-6e7e-464b-84b1-100e44709e2f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735117882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Upstairs opts: Upstairs UUID: 4e99ac97-e5aa-406b-b83e-1cceb43b3c5e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735152729Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Crucible stats registered with UUID: 4e99ac97-e5aa-406b-b83e-1cceb43b3c5e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735163338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:47.735172507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735322974Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735352401Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73536311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73548585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735509978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"4e99ac97-e5aa-406b-b83e-1cceb43b3c5e active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735528356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735537225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735546774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735556144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735565483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735575192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735584751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:56026)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735645936Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50331)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735771015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54659)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.735789993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736311518Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736375552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736386881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736618911Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736641449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[0]R flush_numbers[0..12]: [68, 63, 76, 70, 70, 76, 76, 68, 76, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736734371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[0]R generation[0..12]: [42, 39, 45, 43, 43, 45, 45, 42, 45, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73674582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736756809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[1]R flush_numbers[0..12]: [68, 63, 76, 70, 70, 76, 76, 68, 76, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736767198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[1]R generation[0..12]: [42, 39, 45, 43, 43, 45, 45, 42, 45, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736777467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736788276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[2]R flush_numbers[0..12]: [68, 63, 76, 70, 70, 76, 76, 68, 76, 67, 68, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736798535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[2]R generation[0..12]: [42, 39, 45, 43, 43, 45, 45, 42, 45, 41, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736808745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736819594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736828743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736838222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Next flush: 77","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736847271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73685646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"":"downstairs","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.73686594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736874379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"0","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736885338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"1","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736894797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"client":"2","":"downstairs","session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736904466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736913465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"4e99ac97-e5aa-406b-b83e-1cceb43b3c5e is now active with session: ef2f87ce-6e7e-464b-84b1-100e44709e2f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736923155Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736931624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"session_id":"ef2f87ce-6e7e-464b-84b1-100e44709e2f"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736940453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Activated sub_volume 4e99ac97-e5aa-406b-b83e-1cceb43b3c5e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736948482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.736956562Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.737267215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} 01/30 Write block 506 len 32768 data: 6 7 5 6 7 6 8 8 02/30 Write block 363 len 8192 data: 9 9 03/30 Read block 78 len 28672 04/30 Write block 428 len 20480 data: 6 6 7 5 5 05/30 Read block 242 len 4096 06/30 Write block 499 len 8192 data: 3 2 07/30 Write block 339 len 32768 data: 5 6 5 4 3 2 3 3 08/30 Read block 359 len 8192 09/30 Read block 407 len 8192 10/30 Flush 11/30 Write block 90 len 16384 data: 5 3 3 3 12/30 Read block 398 len 36864 13/30 Read block 474 len 20480 14/30 Read block 205 len 20480 15/30 Write block 279 len 32768 data: 8 7 6 5 5 5 4 4 16/30 Write block 247 len 4096 data: 6 17/30 Read block 502 len 36864 18/30 Read block 533 len 40960 19/30 Flush 20/30 Write block 555 len 32768 data: 4 3 3 2 3 3 5 6 21/30 Read block 498 len 32768 22/30 Write block 221 len 32768 data: 3 2 3 3 3 5 6 7 23/30 Flush 24/30 Write block 50 len 8192 data: 5 5 25/30 Write block 287 len 40960 data: 4 3 2 2 2 3 5 6 8 8 26/30 Read block 126 len 20480 27/30 Read block 288 len 16384 28/30 Write block 218 len 20480 data: 5 6 4 4 3 29/30 Write block 403 len 32768 data: 8 8 7 5 5 3 3 3 30/30 Read block 13 len 28672 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Sent Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:47.74650641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:47.746671295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1320,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 45 45 45 76 76 76 T F T <--- 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 43 46 78 70 78 F F F <--- 5 100-119 45 45 45 76 76 76 F F F 6 120-139 45 45 45 76 76 76 F F F 7 140-159 42 42 42 68 68 68 F F F 8 160-179 45 45 45 76 76 76 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 42 42 42 68 68 68 T F T <--- 11 220-239 46 42 46 79 68 79 T F T <--- 12 240-259 46 45 46 78 76 78 F F F <--- 13 260-279 46 45 46 78 76 78 F F F <--- 14 280-299 46 43 46 78 70 78 T F T <--- 15 300-319 44 44 44 72 72 72 F F F 16 320-339 46 44 46 77 75 77 F F F <--- 17 340-359 46 36 46 77 58 77 F F F <--- 18 360-379 46 44 46 77 72 77 F F F <--- 19 380-399 42 42 42 68 68 68 F F F 20 400-419 41 41 41 67 67 67 T F T <--- 21 420-439 46 43 46 77 70 77 F F F <--- 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 42 46 77 69 77 F F F <--- 25 500-519 46 42 46 77 68 77 F F F <--- 26 520-539 43 43 43 70 70 70 F F F 27 540-559 46 34 46 79 54 79 F F F <--- 28 560-579 46 42 46 79 68 79 F F F <--- 29 580-599 44 44 44 72 72 72 F F F Max gen: 46, Max flush: 79 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 39 39 63 63 63 F F F 2 040-059 47 47 47 80 80 80 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 46 46 78 78 78 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 45 45 45 76 76 76 F F F 7 140-159 42 42 42 68 68 68 F F F 8 160-179 45 45 45 76 76 76 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 47 47 47 80 80 80 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 47 47 47 80 80 80 F F F 15 300-319 44 44 44 72 72 72 F F F 16 320-339 46 46 46 77 77 77 F F F 17 340-359 46 46 46 77 77 77 F F F 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 47 47 47 80 80 80 F F F 21 420-439 46 46 46 77 77 77 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 46 46 77 77 77 F F F 25 500-519 46 46 46 77 77 77 F F F 26 520-539 43 43 43 70 70 70 F F F 27 540-559 46 46 46 79 79 79 F F F 28 560-579 46 46 46 79 79 79 F F F 29 580-599 44 44 44 72 72 72 F F F Max gen: 47, Max flush: 80 Begin loop 24 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.003448802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:52.010663663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.010719118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.017671073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.01793539Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.017964567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.017975126Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.017986455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Crucible ff06e3c2-9605-4467-937b-51e4bc8478b5 has session id: b037b041-b49e-43d9-b284-3156c39d9587","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018303588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Upstairs opts: Upstairs UUID: ff06e3c2-9605-4467-937b-51e4bc8478b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018331635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Crucible stats registered with UUID: ff06e3c2-9605-4467-937b-51e4bc8478b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018342424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:52.018351514Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018461884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018478292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018488192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018645518Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018663896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"ff06e3c2-9605-4467-937b-51e4bc8478b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018690144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018699323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018708852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018718532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018730001Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.01873982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018749789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"ds_connection connected from Ok(127.0.0.1:50807)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018800854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"ds_connection connected from Ok(127.0.0.1:41239)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.018826542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"ds_connection connected from Ok(127.0.0.1:61121)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.01884599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019486365Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019528311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019549969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019799267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019842784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[0]R flush_numbers[0..12]: [68, 63, 80, 70, 78, 76, 76, 68, 76, 67, 80, 80]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019871451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[0]R generation[0..12]: [42, 39, 47, 43, 46, 45, 45, 42, 45, 41, 47, 47]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.01988266Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019893639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[1]R flush_numbers[0..12]: [68, 63, 80, 70, 78, 76, 76, 68, 76, 67, 80, 80]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019903828Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[1]R generation[0..12]: [42, 39, 47, 43, 46, 45, 45, 42, 45, 41, 47, 47]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019916437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019927276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[2]R flush_numbers[0..12]: [68, 63, 80, 70, 78, 76, 76, 68, 76, 67, 80, 80]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019937565Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[2]R generation[0..12]: [42, 39, 47, 43, 46, 45, 45, 42, 45, 41, 47, 47]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019947724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019958554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019967893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019977442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Next flush: 81","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.019986701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020008179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"":"downstairs","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020017968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020026698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"0","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020036187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"1","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020045536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"client":"2","":"downstairs","session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020055035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020063884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"ff06e3c2-9605-4467-937b-51e4bc8478b5 is now active with session: b037b041-b49e-43d9-b284-3156c39d9587","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020074933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020083553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"session_id":"b037b041-b49e-43d9-b284-3156c39d9587"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020092522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Activated sub_volume ff06e3c2-9605-4467-937b-51e4bc8478b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.020100731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.02010878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.02045459Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} 01/30 Write block 543 len 12288 data: 2 2 2 02/30 Read block 467 len 20480 03/30 Write block 571 len 24576 data: 3 3 3 3 3 3 04/30 Write block 330 len 4096 data: 5 05/30 Write block 34 len 8192 data: 5 5 06/30 Read block 351 len 12288 07/30 Read block 520 len 8192 08/30 Read block 409 len 4096 09/30 Flush 10/30 Write block 141 len 28672 data: 5 4 3 4 4 3 4 11/30 Read block 467 len 4096 12/30 Write block 136 len 28672 data: 5 5 5 6 6 6 5 13/30 Flush 14/30 Flush 15/30 Write block 301 len 40960 data: 7 8 9 7 6 6 6 6 7 7 16/30 Read block 564 len 40960 17/30 Read block 373 len 16384 18/30 Read block 462 len 4096 19/30 Write block 160 len 24576 data: 3 3 3 2 3 3 20/30 Write block 153 len 40960 data: 7 7 6 5 6 6 5 4 4 4 21/30 Read block 426 len 24576 22/30 Read block 362 len 4096 23/30 Read block 29 len 16384 24/30 Write block 234 len 8192 data: 5 7 25/30 Write block 410 len 28672 data: 4 3 3 4 4 4 5 26/30 Read block 592 len 32768 27/30 Read block 282 len 32768 28/30 Write block 513 len 36864 data: 9 8 8 7 6 5 6 5 6 29/30 Read block 139 len 12288 30/30 Write block 350 len 12288 data: 6 5 5 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 2 Done Done Done false 1005 Acked Read 3 Done Done Done false 1006 Acked Read 2 Done Done Done false 1007 Acked Read 1 Done Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 22 1 1 24 Done 8 29 29 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:52.028978568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:52.029116846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1328,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 39 48 48 63 81 81 T F F <--- 2 040-059 47 47 47 80 80 80 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 46 46 78 78 78 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 45 48 48 76 82 82 F F F <--- 7 140-159 42 48 48 68 82 82 F T T <--- 8 160-179 45 45 45 76 76 76 F T T <--- 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 47 47 47 80 80 80 F T T <--- 12 240-259 46 46 46 78 78 78 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 47 47 47 80 80 80 F F F 15 300-319 44 44 44 72 72 72 F T T <--- 16 320-339 46 48 48 77 81 81 T F F <--- 17 340-359 46 46 46 77 77 77 F T T <--- 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 47 47 47 80 80 80 F T T <--- 21 420-439 46 46 46 77 77 77 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 46 46 77 77 77 F F F 25 500-519 46 46 46 77 77 77 F T T <--- 26 520-539 43 43 43 70 70 70 F T T <--- 27 540-559 46 48 48 79 81 81 T F F <--- 28 560-579 46 48 48 79 81 81 T F F <--- 29 580-599 44 44 44 72 72 72 F F F Max gen: 48, Max flush: 82 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 48 48 48 81 81 81 F F F 2 040-059 47 47 47 80 80 80 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 46 46 78 78 78 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 48 48 48 82 82 82 F F F 7 140-159 49 49 49 83 83 83 F F F 8 160-179 49 49 49 83 83 83 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 49 49 49 83 83 83 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 47 47 47 80 80 80 F F F 15 300-319 49 49 49 83 83 83 F F F 16 320-339 48 48 48 81 81 81 F F F 17 340-359 49 49 49 83 83 83 F F F 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 49 49 49 83 83 83 F F F 21 420-439 46 46 46 77 77 77 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 46 46 77 77 77 F F F 25 500-519 49 49 49 83 83 83 F F F 26 520-539 49 49 49 83 83 83 F F F 27 540-559 48 48 48 81 81 81 F F F 28 560-579 48 48 48 81 81 81 F F F 29 580-599 44 44 44 72 72 72 F F F Max gen: 49, Max flush: 83 Begin loop 25 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.283768168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292111501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292391756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292416304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292425363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292437422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Crucible 15d5603c-c52b-4b31-8c88-ff30afda72f5 has session id: c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292735606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Upstairs opts: Upstairs UUID: 15d5603c-c52b-4b31-8c88-ff30afda72f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292751585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Crucible stats registered with UUID: 15d5603c-c52b-4b31-8c88-ff30afda72f5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292761594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:56.292773323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292914371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292930839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.292941169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293132712Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293143641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"15d5603c-c52b-4b31-8c88-ff30afda72f5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293160019Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293169179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293178958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293188827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293198686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293208885Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293233663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"ds_connection connected from Ok(127.0.0.1:52262)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293276239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"ds_connection connected from Ok(127.0.0.1:65487)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293315146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"ds_connection connected from Ok(127.0.0.1:39100)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293328055Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293895625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.293920673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294078699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.29418136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294197339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[0]R flush_numbers[0..12]: [68, 81, 80, 70, 78, 76, 82, 83, 83, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294351196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[0]R generation[0..12]: [42, 48, 47, 43, 46, 45, 48, 49, 49, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294362175Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294373204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[1]R flush_numbers[0..12]: [68, 81, 80, 70, 78, 76, 82, 83, 83, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294383563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[1]R generation[0..12]: [42, 48, 47, 43, 46, 45, 48, 49, 49, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294393792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294404571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[2]R flush_numbers[0..12]: [68, 81, 80, 70, 78, 76, 82, 83, 83, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.29441482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[2]R generation[0..12]: [42, 48, 47, 43, 46, 45, 48, 49, 49, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294425109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294435938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294446667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294456197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Next flush: 84","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294465146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294474255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"":"downstairs","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294483924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294492573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"0","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294502013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"1","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294511382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"client":"2","":"downstairs","session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294521201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.29453027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"15d5603c-c52b-4b31-8c88-ff30afda72f5 is now active with session: c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294540009Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294548678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"session_id":"c9a98aa3-412c-45fb-8cd0-ac2f89f3ffda"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294557678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Activated sub_volume 15d5603c-c52b-4b31-8c88-ff30afda72f5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294565637Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294573576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.294934935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} 01/30 Write block 300 len 36864 data: 8 8 9 10 8 7 7 7 7 02/30 Write block 154 len 40960 data: 8 7 6 7 7 6 5 5 5 3 03/30 Read block 110 len 4096 04/30 Read block 285 len 12288 05/30 Read block 5 len 32768 06/30 Read block 25 len 40960 07/30 Write block 268 len 40960 data: 5 4 4 4 5 6 6 5 5 6 08/30 Read block 281 len 32768 09/30 Write block 129 len 36864 data: 6 7 7 6 5 6 5 6 6 10/30 Read block 119 len 40960 11/30 Read block 254 len 8192 12/30 Read block 433 len 4096 13/30 Write block 291 len 36864 data: 3 4 6 7 9 9 8 8 10 14/30 Read block 174 len 20480 15/30 Read block 546 len 40960 16/30 Read block 561 len 36864 17/30 Read block 526 len 4096 18/30 Write block 534 len 8192 data: 5 4 19/30 Read block 68 len 24576 20/30 Write block 343 len 24576 data: 4 3 4 4 3 3 21/30 Write block 558 len 28672 data: 3 4 4 6 7 5 5 22/30 Read block 331 len 12288 23/30 Write block 272 len 28672 data: 6 7 7 6 6 7 9 24/30 Read block 488 len 12288 25/30 Flush 26/30 Write block 418 len 4096 data: 4 27/30 Write block 418 len 28672 data: 5 2 3 3 3 3 3 28/30 Read block 430 len 4096 29/30 Write block 34 len 32768 data: 6 6 5 4 3 3 2 3 30/30 Write block 309 len 32768 data: 8 8 6 5 5 6 5 5 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 10 Done Sent Done false 1002 Acked Read 1 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:29:56.303933481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:29:56.304097847Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1336,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 48 48 48 81 81 81 T F T <--- 2 040-059 47 47 47 80 80 80 T F T <--- 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 46 46 78 78 78 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 50 48 50 84 82 84 F F F <--- 7 140-159 50 49 50 84 83 84 F F F <--- 8 160-179 50 49 50 84 83 84 F F F <--- 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 49 49 49 83 83 83 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 50 46 50 84 78 84 F F F <--- 14 280-299 50 47 50 84 80 84 F F F <--- 15 300-319 50 49 50 84 83 84 T T T <--- 16 320-339 48 48 48 81 81 81 F F F 17 340-359 50 49 50 84 83 84 F F F <--- 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 49 49 49 83 83 83 T F T <--- 21 420-439 46 46 46 77 77 77 T F T <--- 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 46 46 77 77 77 F F F 25 500-519 49 49 49 83 83 83 F F F 26 520-539 50 49 50 84 83 84 F F F <--- 27 540-559 50 48 50 84 81 84 F F F <--- 28 560-579 50 48 50 84 81 84 F F F <--- 29 580-599 44 44 44 72 72 72 F F F Max gen: 50, Max flush: 84 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 51 51 51 85 85 85 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 46 46 78 78 78 F F F 5 100-119 45 45 45 76 76 76 F F F 6 120-139 50 50 50 84 84 84 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 49 49 49 83 83 83 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 50 50 50 84 84 84 F F F 14 280-299 50 50 50 84 84 84 F F F 15 300-319 51 51 51 85 85 85 F F F 16 320-339 48 48 48 81 81 81 F F F 17 340-359 50 50 50 84 84 84 F F F 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 51 51 51 85 85 85 F F F 21 420-439 51 51 51 85 85 85 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 46 46 77 77 77 F F F 25 500-519 49 49 49 83 83 83 F F F 26 520-539 50 50 50 84 84 84 F F F 27 540-559 50 50 50 84 84 84 F F F 28 560-579 50 50 50 84 84 84 F F F 29 580-599 44 44 44 72 72 72 F F F Max gen: 51, Max flush: 85 Begin loop 26 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.555728152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:00.562895267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.562942623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570019657Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570262975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570296033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570304412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570315841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Crucible f171ee9d-0760-4e29-91c2-bf0e17ecf1c8 has session id: c83c5286-2a21-4f65-8847-bf68480ea527","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570650432Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Upstairs opts: Upstairs UUID: f171ee9d-0760-4e29-91c2-bf0e17ecf1c8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570683509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Crucible stats registered with UUID: f171ee9d-0760-4e29-91c2-bf0e17ecf1c8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570694188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:00.570703227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570863473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570933997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.570960425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571029809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571044867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"f171ee9d-0760-4e29-91c2-bf0e17ecf1c8 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571062486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571071185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571080614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571090073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571099413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571109162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571118721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"ds_connection connected from Ok(127.0.0.1:40488)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571153588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"ds_connection connected from Ok(127.0.0.1:46632)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571276677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"ds_connection connected from Ok(127.0.0.1:40365)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571302275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571873405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571891724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.571902573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.57215814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572189968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[0]R flush_numbers[0..12]: [68, 85, 85, 70, 78, 76, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572238993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[0]R generation[0..12]: [42, 51, 51, 43, 46, 45, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572249922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572260781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[1]R flush_numbers[0..12]: [68, 85, 85, 70, 78, 76, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572271141Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[1]R generation[0..12]: [42, 51, 51, 43, 46, 45, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.57228117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572291669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[2]R flush_numbers[0..12]: [68, 85, 85, 70, 78, 76, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572301888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[2]R generation[0..12]: [42, 51, 51, 43, 46, 45, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572311987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572322696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572331785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572341174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Next flush: 86","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572350154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572359203Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"":"downstairs","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572368622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572378451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"0","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.57238779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"1","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.5723973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"client":"2","":"downstairs","session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572407129Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572415958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"f171ee9d-0760-4e29-91c2-bf0e17ecf1c8 is now active with session: c83c5286-2a21-4f65-8847-bf68480ea527","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572425597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572434236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"session_id":"c83c5286-2a21-4f65-8847-bf68480ea527"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572443076Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Activated sub_volume f171ee9d-0760-4e29-91c2-bf0e17ecf1c8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572450855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572458684Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.572798615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} 01/30 Write block 558 len 24576 data: 4 5 5 7 8 6 02/30 Read block 177 len 20480 03/30 Read block 262 len 4096 04/30 Read block 523 len 40960 05/30 Flush 06/30 Write block 582 len 8192 data: 4 5 07/30 Read block 21 len 16384 08/30 Read block 51 len 36864 09/30 Read block 208 len 36864 10/30 Flush 11/30 Write block 481 len 12288 data: 6 6 6 12/30 Read block 216 len 28672 13/30 Read block 0 len 40960 14/30 Write block 302 len 28672 data: 10 11 9 8 8 8 8 15/30 Read block 536 len 16384 16/30 Read block 199 len 12288 17/30 Read block 94 len 28672 18/30 Write block 279 len 12288 data: 9 8 7 19/30 Write block 555 len 4096 data: 5 20/30 Write block 110 len 24576 data: 6 3 4 5 4 5 21/30 Write block 88 len 8192 data: 5 5 22/30 Flush 23/30 Write block 344 len 36864 data: 4 5 5 4 4 5 7 6 6 24/30 Read block 363 len 4096 25/30 Write block 517 len 16384 data: 7 6 7 6 26/30 Read block 75 len 36864 27/30 Read block 214 len 12288 28/30 Read block 45 len 12288 29/30 Read block 63 len 40960 30/30 Read block 39 len 4096 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Read 10 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:00.581526724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:00.581696329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1344,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 51 51 51 85 85 85 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 46 52 52 78 88 88 F F F <--- 5 100-119 45 52 52 76 88 88 F F F <--- 6 120-139 50 50 50 84 84 84 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 49 49 49 83 83 83 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 50 52 52 84 88 88 F F F <--- 14 280-299 50 52 52 84 88 88 F F F <--- 15 300-319 51 52 52 85 88 88 F F F <--- 16 320-339 48 48 48 81 81 81 F F F 17 340-359 50 50 50 84 84 84 F T T <--- 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 51 51 51 85 85 85 F F F 21 420-439 51 51 51 85 85 85 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 46 52 52 77 88 88 F F F <--- 25 500-519 49 49 49 83 83 83 F T T <--- 26 520-539 50 50 50 84 84 84 F T T <--- 27 540-559 50 52 52 84 88 88 T F F <--- 28 560-579 50 52 52 84 86 86 T F F <--- 29 580-599 44 52 52 72 87 87 F F F <--- Max gen: 52, Max flush: 88 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 68 68 68 F F F 1 020-039 51 51 51 85 85 85 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 52 52 52 88 88 88 F F F 5 100-119 52 52 52 88 88 88 F F F 6 120-139 50 50 50 84 84 84 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 47 47 47 80 80 80 F F F 11 220-239 49 49 49 83 83 83 F F F 12 240-259 46 46 46 78 78 78 F F F 13 260-279 52 52 52 88 88 88 F F F 14 280-299 52 52 52 88 88 88 F F F 15 300-319 52 52 52 88 88 88 F F F 16 320-339 48 48 48 81 81 81 F F F 17 340-359 53 53 53 89 89 89 F F F 18 360-379 46 46 46 77 77 77 F F F 19 380-399 42 42 42 68 68 68 F F F 20 400-419 51 51 51 85 85 85 F F F 21 420-439 51 51 51 85 85 85 F F F 22 440-459 44 44 44 74 74 74 F F F 23 460-479 39 39 39 63 63 63 F F F 24 480-499 52 52 52 88 88 88 F F F 25 500-519 53 53 53 89 89 89 F F F 26 520-539 53 53 53 89 89 89 F F F 27 540-559 52 52 52 88 88 88 F F F 28 560-579 52 52 52 86 86 86 F F F 29 580-599 52 52 52 87 87 87 F F F Max gen: 53, Max flush: 89 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.835516253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.84382224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844102955Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844128383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844137042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844148291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Crucible 251a5e61-c42b-42e3-b7f1-4684259d4117 has session id: 09e64b03-9247-4212-891e-3edf2fd7072f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844448045Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Upstairs opts: Upstairs UUID: 251a5e61-c42b-42e3-b7f1-4684259d4117, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.84450396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Crucible stats registered with UUID: 251a5e61-c42b-42e3-b7f1-4684259d4117","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.84451431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:04.844523099Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844656347Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844685675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844696214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844845601Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844862119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"251a5e61-c42b-42e3-b7f1-4684259d4117 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844897336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844906595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844916085Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844925584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844935013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844944682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844956591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"ds_connection connected from Ok(127.0.0.1:48273)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.844999157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"ds_connection connected from Ok(127.0.0.1:52528)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845013736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"ds_connection connected from Ok(127.0.0.1:64654)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845109878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"io task","client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845631002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845649321Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845686557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845950374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.845973012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[0]R flush_numbers[0..12]: [68, 85, 85, 70, 88, 88, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.84600368Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[0]R generation[0..12]: [42, 51, 51, 43, 52, 52, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846014969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846025978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[1]R flush_numbers[0..12]: [68, 85, 85, 70, 88, 88, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846036257Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[1]R generation[0..12]: [42, 51, 51, 43, 52, 52, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846046596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846057375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[2]R flush_numbers[0..12]: [68, 85, 85, 70, 88, 88, 84, 84, 84, 67, 80, 83]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846069644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[2]R generation[0..12]: [42, 51, 51, 43, 52, 52, 50, 50, 50, 41, 47, 49]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846079893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846090592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846099801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846109201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.84611859Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846127889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"":"downstairs","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846137488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846146037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"0","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846155587Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"1","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846165046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"client":"2","":"downstairs","session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846174595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846183444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"251a5e61-c42b-42e3-b7f1-4684259d4117 is now active with session: 09e64b03-9247-4212-891e-3edf2fd7072f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846193193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846201733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"session_id":"09e64b03-9247-4212-891e-3edf2fd7072f"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846211812Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Activated sub_volume 251a5e61-c42b-42e3-b7f1-4684259d4117","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846219681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.8462275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.846521605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} 01/30 Write block 3 len 32768 data: 3 3 4 4 4 5 6 5 02/30 Read block 220 len 32768 03/30 Read block 49 len 8192 04/30 Write block 363 len 28672 data: 10 10 10 8 9 9 9 05/30 Write block 219 len 8192 data: 7 5 06/30 Read block 62 len 28672 07/30 Read block 23 len 24576 08/30 Write block 441 len 24576 data: 6 8 7 9 7 9 09/30 Write block 139 len 4096 data: 7 10/30 Read block 173 len 28672 11/30 Read block 450 len 28672 12/30 Write block 88 len 8192 data: 6 6 13/30 Flush 14/30 Write block 392 len 16384 data: 9 6 6 5 15/30 Write block 477 len 40960 data: 7 8 6 4 7 7 7 6 5 7 16/30 Read block 175 len 24576 17/30 Write block 512 len 36864 data: 9 10 9 9 8 8 7 8 7 18/30 Read block 206 len 40960 19/30 Read block 327 len 16384 20/30 Read block 359 len 28672 21/30 Write block 435 len 24576 data: 3 4 5 5 5 6 22/30 Write block 265 len 32768 data: 5 6 6 6 5 5 5 7 23/30 Write block 234 len 40960 data: 6 8 6 7 6 5 5 5 4 3 24/30 Read block 406 len 16384 25/30 Flush 26/30 Write block 296 len 4096 data: 10 27/30 Write block 479 len 20480 data: 7 5 8 8 8 28/30 Read block 25 len 40960 29/30 Write block 131 len 28672 data: 8 7 6 7 6 7 7 30/30 Read block 95 len 32768 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:04.855971751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:04.856170574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1355,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 42 90 90 68 F F T <--- 1 020-039 51 51 51 85 85 85 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 54 54 52 90 90 88 F F F <--- 5 100-119 52 52 52 88 88 88 F F F 6 120-139 54 54 50 90 90 84 T T F <--- 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 54 54 47 90 90 80 F F F <--- 11 220-239 54 54 49 91 91 83 F F F <--- 12 240-259 54 54 46 91 91 78 F F F <--- 13 260-279 54 54 52 91 91 88 F F F <--- 14 280-299 52 52 52 88 88 88 T T F <--- 15 300-319 52 52 52 88 88 88 F F F 16 320-339 48 48 48 81 81 81 F F F 17 340-359 53 53 53 89 89 89 F F F 18 360-379 54 54 46 90 90 77 F F F <--- 19 380-399 54 54 42 91 91 68 F F F <--- 20 400-419 51 51 51 85 85 85 F F F 21 420-439 54 54 51 91 91 85 F F F <--- 22 440-459 54 54 44 91 91 74 F F F <--- 23 460-479 54 54 39 91 91 63 T T F <--- 24 480-499 54 54 52 91 91 88 T T F <--- 25 500-519 54 54 53 91 91 89 F F F <--- 26 520-539 54 54 53 91 91 89 F F F <--- 27 540-559 52 52 52 88 88 88 F F F 28 560-579 52 52 52 86 86 86 F F F 29 580-599 52 52 52 87 87 87 F F F Max gen: 54, Max flush: 91 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 90 90 90 F F F 1 020-039 51 51 51 85 85 85 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 F F F 4 080-099 54 54 54 90 90 90 F F F 5 100-119 52 52 52 88 88 88 F F F 6 120-139 55 55 55 92 92 92 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 41 41 41 67 67 67 F F F 10 200-219 54 54 54 90 90 90 F F F 11 220-239 54 54 54 91 91 91 F F F 12 240-259 54 54 54 91 91 91 F F F 13 260-279 54 54 54 91 91 91 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 52 52 52 88 88 88 F F F 16 320-339 48 48 48 81 81 81 F F F 17 340-359 53 53 53 89 89 89 F F F 18 360-379 54 54 54 90 90 90 F F F 19 380-399 54 54 54 91 91 91 F F F 20 400-419 51 51 51 85 85 85 F F F 21 420-439 54 54 54 91 91 91 F F F 22 440-459 54 54 54 91 91 91 F F F 23 460-479 55 55 55 92 92 92 F F F 24 480-499 55 55 55 92 92 92 F F F 25 500-519 54 54 54 91 91 91 F F F 26 520-539 54 54 54 91 91 91 F F F 27 540-559 52 52 52 88 88 88 F F F 28 560-579 52 52 52 86 86 86 F F F 29 580-599 52 52 52 87 87 87 F F F Max gen: 55, Max flush: 92 Begin loop 28 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.110955334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119202215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119512648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119546325Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119555434Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119566793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Crucible 209a2ef5-05e1-447b-9875-2247137f87e0 has session id: be630b03-58a8-421a-9996-e66cc1525177","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119934371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Upstairs opts: Upstairs UUID: 209a2ef5-05e1-447b-9875-2247137f87e0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119958289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Crucible stats registered with UUID: 209a2ef5-05e1-447b-9875-2247137f87e0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.119968198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:09.119976988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120115826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120134094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120144063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.12029623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120319908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"209a2ef5-05e1-447b-9875-2247137f87e0 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120338176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120346825Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120356295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120365674Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120375103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120384772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120394571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"ds_connection connected from Ok(127.0.0.1:46449)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120481604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"ds_connection connected from Ok(127.0.0.1:60022)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.12052156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"ds_connection connected from Ok(127.0.0.1:51164)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.120550528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"io task","client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121139126Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121189172Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121202311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121456739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121498065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[0]R flush_numbers[0..12]: [90, 85, 85, 70, 90, 88, 92, 84, 84, 67, 90, 91]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121527483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[0]R generation[0..12]: [54, 51, 51, 43, 54, 52, 55, 50, 50, 41, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121538462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121549311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[1]R flush_numbers[0..12]: [90, 85, 85, 70, 90, 88, 92, 84, 84, 67, 90, 91]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.12155967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[1]R generation[0..12]: [54, 51, 51, 43, 54, 52, 55, 50, 50, 41, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121569899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121580658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[2]R flush_numbers[0..12]: [90, 85, 85, 70, 90, 88, 92, 84, 84, 67, 90, 91]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121590937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[2]R generation[0..12]: [54, 51, 51, 43, 54, 52, 55, 50, 50, 41, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121601096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121611755Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121620834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121630314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Next flush: 93","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121641023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121650232Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"":"downstairs","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121659681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.12166834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"0","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121677869Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"1","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121687469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"client":"2","":"downstairs","session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121697168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121706157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"209a2ef5-05e1-447b-9875-2247137f87e0 is now active with session: be630b03-58a8-421a-9996-e66cc1525177","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121715966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121724525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"session_id":"be630b03-58a8-421a-9996-e66cc1525177"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121733355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Activated sub_volume 209a2ef5-05e1-447b-9875-2247137f87e0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121741164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.121749033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.122087554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} 01/30 Write block 469 len 28672 data: 6 5 4 4 4 8 8 02/30 Write block 444 len 20480 data: 10 8 10 8 9 03/30 Write block 232 len 4096 data: 6 04/30 Flush 05/30 Write block 415 len 28672 data: 5 6 4 6 3 4 4 06/30 Write block 391 len 8192 data: 7 10 07/30 Read block 550 len 36864 08/30 Read block 484 len 28672 09/30 Read block 381 len 20480 10/30 Write block 322 len 28672 data: 9 9 9 6 6 6 6 11/30 Write block 483 len 8192 data: 9 7 12/30 Flush 13/30 Flush 14/30 Write block 259 len 8192 data: 5 5 15/30 Read block 153 len 28672 16/30 Write block 33 len 20480 data: 5 7 7 6 5 17/30 Write block 183 len 40960 data: 5 5 4 4 4 4 3 3 4 4 18/30 Read block 178 len 12288 19/30 Write block 571 len 20480 data: 4 4 4 4 4 20/30 Write block 585 len 40960 data: 8 10 8 8 10 9 9 8 7 6 21/30 Flush 22/30 Write block 258 len 36864 data: 6 6 6 8 6 4 4 6 7 23/30 Write block 333 len 32768 data: 4 4 4 4 4 5 6 7 24/30 Write block 75 len 24576 data: 4 5 4 4 3 4 25/30 Read block 535 len 12288 26/30 Read block 28 len 16384 27/30 Write block 511 len 12288 data: 7 10 11 28/30 Write block 312 len 36864 data: 6 6 7 6 6 6 6 8 8 29/30 Read block 129 len 24576 30/30 Read block 202 len 20480 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 7 Done Done Done false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 5 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 25 25 Done 26 26 1 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:09.130862519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:09.131046263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1363,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 90 90 90 F F F 1 020-039 56 56 51 96 96 85 F F F <--- 2 040-059 51 51 51 85 85 85 F F F 3 060-079 43 43 43 70 70 70 T T F <--- 4 080-099 54 54 54 90 90 90 T T F <--- 5 100-119 52 52 52 88 88 88 F F F 6 120-139 55 55 55 92 92 92 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 41 96 96 67 F F F <--- 10 200-219 54 54 54 90 90 90 F F F 11 220-239 56 56 56 93 93 93 F F F 12 240-259 56 56 54 96 96 91 T T F <--- 13 260-279 56 56 54 96 96 91 T T F <--- 14 280-299 55 55 55 92 92 92 F F F 15 300-319 52 52 52 88 88 88 T T F <--- 16 320-339 56 56 48 94 94 81 T T F <--- 17 340-359 53 53 53 89 89 89 T T F <--- 18 360-379 54 54 54 90 90 90 F F F 19 380-399 56 56 54 94 94 91 F F F <--- 20 400-419 56 56 51 94 94 85 F F T <--- 21 420-439 56 56 54 94 94 91 F F T <--- 22 440-459 56 56 56 93 93 93 F F F 23 460-479 56 56 56 93 93 93 F F F 24 480-499 56 56 55 94 94 92 F F F <--- 25 500-519 54 54 54 91 91 91 T T F <--- 26 520-539 54 54 54 91 91 91 F F F 27 540-559 52 52 52 88 88 88 F F F 28 560-579 56 56 52 96 96 86 F F F <--- 29 580-599 56 56 52 96 96 87 F F F <--- Max gen: 56, Max flush: 96 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 90 90 90 F F F 1 020-039 56 56 56 96 96 96 F F F 2 040-059 51 51 51 85 85 85 F F F 3 060-079 57 57 57 97 97 97 F F F 4 080-099 57 57 57 97 97 97 F F F 5 100-119 52 52 52 88 88 88 F F F 6 120-139 55 55 55 92 92 92 F F F 7 140-159 50 50 50 84 84 84 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 56 96 96 96 F F F 10 200-219 54 54 54 90 90 90 F F F 11 220-239 56 56 56 93 93 93 F F F 12 240-259 57 57 57 97 97 97 F F F 13 260-279 57 57 57 97 97 97 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 57 57 57 97 97 97 F F F 17 340-359 57 57 57 97 97 97 F F F 18 360-379 54 54 54 90 90 90 F F F 19 380-399 56 56 56 94 94 94 F F F 20 400-419 56 56 56 94 94 94 F F F 21 420-439 56 56 56 94 94 94 F F F 22 440-459 56 56 56 93 93 93 F F F 23 460-479 56 56 56 93 93 93 F F F 24 480-499 56 56 56 94 94 94 F F F 25 500-519 57 57 57 97 97 97 F F F 26 520-539 54 54 54 91 91 91 F F F 27 540-559 52 52 52 88 88 88 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 56 56 56 96 96 96 F F F Max gen: 57, Max flush: 97 Begin loop 29 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.391998275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400264915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400532082Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400562299Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400572628Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400584007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Crucible 564b455e-9d51-454c-b068-41a5ac295e93 has session id: e298af5e-02e4-4a66-af5d-09355166f7c1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400910959Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Upstairs opts: Upstairs UUID: 564b455e-9d51-454c-b068-41a5ac295e93, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400939796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Crucible stats registered with UUID: 564b455e-9d51-454c-b068-41a5ac295e93","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.400950455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:13.400959565Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401078644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401095683Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401105572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401268018Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401291786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"564b455e-9d51-454c-b068-41a5ac295e93 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401317623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401326993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401336542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401347231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.40135658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401366499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401376158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:57018)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401402766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:51611)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401437003Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:58335)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401507297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"io task","client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.401999404Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402024972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402091846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402299128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402335305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[0]R flush_numbers[0..12]: [90, 96, 85, 97, 97, 88, 92, 84, 84, 96, 90, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402371922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[0]R generation[0..12]: [54, 56, 51, 57, 57, 52, 55, 50, 50, 56, 54, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402382581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.4023937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[1]R flush_numbers[0..12]: [90, 96, 85, 97, 97, 88, 92, 84, 84, 96, 90, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402406449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[1]R generation[0..12]: [54, 56, 51, 57, 57, 52, 55, 50, 50, 56, 54, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402416698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402427387Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[2]R flush_numbers[0..12]: [90, 96, 85, 97, 97, 88, 92, 84, 84, 96, 90, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402437576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[2]R generation[0..12]: [54, 56, 51, 57, 57, 52, 55, 50, 50, 56, 54, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402447735Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402458244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402467673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402477052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Next flush: 98","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402486162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402495381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"":"downstairs","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.40250506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402513819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"0","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402523388Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"1","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402532918Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"client":"2","":"downstairs","session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402542597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402552966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"564b455e-9d51-454c-b068-41a5ac295e93 is now active with session: e298af5e-02e4-4a66-af5d-09355166f7c1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402562745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402571234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"session_id":"e298af5e-02e4-4a66-af5d-09355166f7c1"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402580043Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Activated sub_volume 564b455e-9d51-454c-b068-41a5ac295e93","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402588053Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402595982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.402904215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} 01/30 Write block 537 len 4096 data: 4 02/30 Write block 138 len 40960 data: 6 8 7 7 6 4 5 5 4 5 03/30 Read block 373 len 16384 04/30 Read block 164 len 36864 05/30 Read block 450 len 20480 06/30 Write block 347 len 24576 data: 5 5 6 8 7 7 07/30 Write block 586 len 32768 data: 11 9 9 11 10 10 9 8 08/30 Read block 422 len 24576 09/30 Write block 15 len 28672 data: 4 4 4 3 4 3 2 10/30 Write block 377 len 16384 data: 11 9 7 7 11/30 Read block 51 len 4096 12/30 Read block 361 len 32768 13/30 Write block 498 len 8192 data: 3 4 14/30 Write block 465 len 8192 data: 6 5 15/30 Read block 327 len 20480 16/30 Read block 140 len 4096 17/30 Write block 211 len 24576 data: 6 6 4 4 5 4 18/30 Write block 326 len 16384 data: 7 7 7 5 19/30 Write block 48 len 40960 data: 8 7 6 6 6 6 7 5 4 4 20/30 Read block 511 len 28672 21/30 Read block 379 len 8192 22/30 Read block 570 len 36864 23/30 Read block 48 len 32768 24/30 Write block 11 len 20480 data: 5 6 5 4 5 25/30 Read block 439 len 16384 26/30 Write block 70 len 12288 data: 4 4 5 27/30 Write block 497 len 24576 data: 3 4 5 3 3 4 28/30 Write block 548 len 40960 data: 2 2 2 2 4 4 4 6 4 4 29/30 Write block 586 len 4096 data: 12 30/30 Read block 4 len 8192 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Read 5 Done Done Done false 1005 Acked Write 6 Done Done Done false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 24 25 Done 29 30 6 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:13.410709905Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:13.41088784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1371,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 90 90 90 T T F <--- 1 020-039 56 56 56 96 96 96 T T F <--- 2 040-059 51 51 51 85 85 85 T T F <--- 3 060-079 57 57 57 97 97 97 T T F <--- 4 080-099 57 57 57 97 97 97 F F F 5 100-119 52 52 52 88 88 88 F F F 6 120-139 55 55 55 92 92 92 T T T 7 140-159 50 50 50 84 84 84 T T T 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 56 96 96 96 F F F 10 200-219 54 54 54 90 90 90 T T F <--- 11 220-239 56 56 56 93 93 93 F F F 12 240-259 57 57 57 97 97 97 F F F 13 260-279 57 57 57 97 97 97 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 57 57 57 97 97 97 T T F <--- 17 340-359 57 57 57 97 97 97 T T T 18 360-379 54 54 54 90 90 90 T T F <--- 19 380-399 56 56 56 94 94 94 T T F <--- 20 400-419 56 56 56 94 94 94 F F F 21 420-439 56 56 56 94 94 94 F F F 22 440-459 56 56 56 93 93 93 F F F 23 460-479 56 56 56 93 93 93 T T F <--- 24 480-499 56 56 56 94 94 94 T T F <--- 25 500-519 57 57 57 97 97 97 T T F <--- 26 520-539 54 54 54 91 91 91 T T T 27 540-559 52 52 52 88 88 88 T T F <--- 28 560-579 56 56 56 96 96 96 F F F 29 580-599 56 56 56 96 96 96 T T F <--- Max gen: 57, Max flush: 97 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 58 58 58 98 98 98 F F F 1 020-039 58 58 58 98 98 98 F F F 2 040-059 58 58 58 98 98 98 F F F 3 060-079 58 58 58 98 98 98 F F F 4 080-099 57 57 57 97 97 97 F F F 5 100-119 52 52 52 88 88 88 F F F 6 120-139 58 58 58 98 98 98 F F F 7 140-159 58 58 58 98 98 98 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 56 96 96 96 F F F 10 200-219 58 58 58 98 98 98 F F F 11 220-239 56 56 56 93 93 93 F F F 12 240-259 57 57 57 97 97 97 F F F 13 260-279 57 57 57 97 97 97 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 58 58 58 98 98 98 F F F 17 340-359 58 58 58 98 98 98 F F F 18 360-379 58 58 58 98 98 98 F F F 19 380-399 58 58 58 98 98 98 F F F 20 400-419 56 56 56 94 94 94 F F F 21 420-439 56 56 56 94 94 94 F F F 22 440-459 56 56 56 93 93 93 F F F 23 460-479 58 58 58 98 98 98 F F F 24 480-499 58 58 58 98 98 98 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 58 58 58 98 98 98 F F F Max gen: 58, Max flush: 98 Begin loop 30 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.677980638Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686143187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686431372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686455279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686463529Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686474908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Crucible 17982bd5-4510-4deb-a943-fe7fffcd953c has session id: 5b5d6b66-8f55-41c9-9878-f33eb2285a3c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686863614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Upstairs opts: Upstairs UUID: 17982bd5-4510-4deb-a943-fe7fffcd953c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686880112Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Crucible stats registered with UUID: 17982bd5-4510-4deb-a943-fe7fffcd953c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.686891971Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:17.686900921Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.6870267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687055777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687066776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687228872Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687244511Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"17982bd5-4510-4deb-a943-fe7fffcd953c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687278898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687288037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687297616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687307145Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687316594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687326324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687335933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:40565)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687378789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:41927)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687389448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46250)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.687401427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"io task","client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688084678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688109635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.68817202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688374742Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688391761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[0]R flush_numbers[0..12]: [98, 98, 98, 98, 97, 88, 98, 98, 84, 96, 98, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688443886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[0]R generation[0..12]: [58, 58, 58, 58, 57, 52, 58, 58, 50, 56, 58, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688454635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688465464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[1]R flush_numbers[0..12]: [98, 98, 98, 98, 97, 88, 98, 98, 84, 96, 98, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688475793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[1]R generation[0..12]: [58, 58, 58, 58, 57, 52, 58, 58, 50, 56, 58, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688485973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688496622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[2]R flush_numbers[0..12]: [98, 98, 98, 98, 97, 88, 98, 98, 84, 96, 98, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688506811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[2]R generation[0..12]: [58, 58, 58, 58, 57, 52, 58, 58, 50, 56, 58, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.68851686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688529419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Max found gen is 59","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688538648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Generation requested: 60 >= found:59","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688548177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Next flush: 99","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688557336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688566696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"":"downstairs","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688576585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688585224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"0","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688595133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"1","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688604862Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"client":"2","":"downstairs","session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688614711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688623961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"17982bd5-4510-4deb-a943-fe7fffcd953c is now active with session: 5b5d6b66-8f55-41c9-9878-f33eb2285a3c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.68863365Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688642209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"session_id":"5b5d6b66-8f55-41c9-9878-f33eb2285a3c"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688651248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Activated sub_volume 17982bd5-4510-4deb-a943-fe7fffcd953c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688659177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688667197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.688964291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} 01/30 Write block 401 len 8192 data: 9 9 02/30 Read block 523 len 20480 03/30 Write block 57 len 12288 data: 5 4 6 04/30 Read block 72 len 40960 05/30 Read block 100 len 8192 06/30 Read block 106 len 28672 07/30 Write block 104 len 20480 data: 6 5 4 5 5 08/30 Read block 422 len 16384 09/30 Write block 144 len 16384 data: 6 6 5 6 10/30 Write block 15 len 20480 data: 6 5 5 4 5 11/30 Read block 5 len 8192 12/30 Read block 104 len 40960 13/30 Read block 122 len 12288 14/30 Read block 558 len 8192 15/30 Read block 384 len 16384 16/30 Read block 10 len 24576 17/30 Read block 381 len 12288 18/30 Read block 102 len 36864 19/30 Write block 65 len 20480 data: 3 3 4 4 4 20/30 Read block 24 len 32768 21/30 Flush 22/30 Write block 325 len 8192 data: 7 8 23/30 Read block 583 len 28672 24/30 Read block 77 len 12288 25/30 Write block 213 len 4096 data: 5 26/30 Read block 184 len 24576 27/30 Write block 456 len 24576 data: 5 4 4 3 5 5 28/30 Read block 30 len 8192 29/30 Write block 254 len 24576 data: 5 5 5 5 7 7 30/30 Read block 40 len 12288 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 4 Done Done Done false 1008 Acked Write 4 Done Done Done false 1009 Acked Write 5 Done Done Done false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 20 21 Done 30 29 10 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:17.697908722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:17.698083296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1379,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 58 99 99 98 F F T <--- 1 020-039 58 58 58 98 98 98 F F F 2 040-059 60 60 58 99 99 98 F F T <--- 3 060-079 60 60 58 99 99 98 F F F <--- 4 080-099 57 57 57 97 97 97 F F F 5 100-119 60 60 52 99 99 88 F F T <--- 6 120-139 58 58 58 98 98 98 F F F 7 140-159 60 60 58 99 99 98 F F T <--- 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 56 96 96 96 F F F 10 200-219 58 58 58 98 98 98 T T F <--- 11 220-239 56 56 56 93 93 93 F F F 12 240-259 57 57 57 97 97 97 T T F <--- 13 260-279 57 57 57 97 97 97 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 58 58 58 98 98 98 T T F <--- 17 340-359 58 58 58 98 98 98 F F F 18 360-379 58 58 58 98 98 98 F F F 19 380-399 58 58 58 98 98 98 F F F 20 400-419 60 60 56 99 99 94 F F T <--- 21 420-439 56 56 56 94 94 94 F F F 22 440-459 56 56 56 93 93 93 T T F <--- 23 460-479 58 58 58 98 98 98 T T F <--- 24 480-499 58 58 58 98 98 98 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 58 58 58 98 98 98 F F F Max gen: 60, Max flush: 99 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 58 58 58 98 98 98 F F F 2 040-059 60 60 60 99 99 99 F F F 3 060-079 60 60 60 99 99 99 F F F 4 080-099 57 57 57 97 97 97 F F F 5 100-119 60 60 60 99 99 99 F F F 6 120-139 58 58 58 98 98 98 F F F 7 140-159 60 60 60 99 99 99 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 56 56 96 96 96 F F F 10 200-219 61 61 61 100 100 100 F F F 11 220-239 56 56 56 93 93 93 F F F 12 240-259 61 61 61 100 100 100 F F F 13 260-279 57 57 57 97 97 97 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 61 61 61 100 100 100 F F F 17 340-359 58 58 58 98 98 98 F F F 18 360-379 58 58 58 98 98 98 F F F 19 380-399 58 58 58 98 98 98 F F F 20 400-419 60 60 60 99 99 99 F F F 21 420-439 56 56 56 94 94 94 F F F 22 440-459 61 61 61 100 100 100 F F F 23 460-479 61 61 61 100 100 100 F F F 24 480-499 58 58 58 98 98 98 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 58 58 58 98 98 98 F F F Max gen: 61, Max flush: 100 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.947344798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:21.954506904Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.954562149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.961351758Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.961608715Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.961634503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.961643122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.961654431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Crucible e6687d78-61a5-4863-8577-2666f39d54b5 has session id: d304ebbb-e534-4d21-aaf8-8f822a610330","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962022109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Upstairs opts: Upstairs UUID: e6687d78-61a5-4863-8577-2666f39d54b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962050967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Crucible stats registered with UUID: e6687d78-61a5-4863-8577-2666f39d54b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962061806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:21.962070895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962194154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962224681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962235261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962363879Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962379448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"e6687d78-61a5-4863-8577-2666f39d54b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962402806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962411875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962421334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962430734Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962440073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962449902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962459571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"ds_connection connected from Ok(127.0.0.1:33436)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962544324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"ds_connection connected from Ok(127.0.0.1:50200)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962606058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"ds_connection connected from Ok(127.0.0.1:37236)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.962617887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"io task","client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963171909Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963200366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963297468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963515339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963540307Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[0]R flush_numbers[0..12]: [99, 98, 99, 99, 97, 99, 98, 99, 84, 96, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963583273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[0]R generation[0..12]: [60, 58, 60, 60, 57, 60, 58, 60, 50, 56, 61, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963594622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963605491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[1]R flush_numbers[0..12]: [99, 98, 99, 99, 97, 99, 98, 99, 84, 96, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.96361589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[1]R generation[0..12]: [60, 58, 60, 60, 57, 60, 58, 60, 50, 56, 61, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963626329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963637248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[2]R flush_numbers[0..12]: [99, 98, 99, 99, 97, 99, 98, 99, 84, 96, 100, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963647468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[2]R generation[0..12]: [60, 58, 60, 60, 57, 60, 58, 60, 50, 56, 61, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963657577Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963668236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963677555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963687174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Next flush: 101","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963696323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963705582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"":"downstairs","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963716502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963725161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"0","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.96373475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"1","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963744249Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"client":"2","":"downstairs","session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963754208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963763277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"e6687d78-61a5-4863-8577-2666f39d54b5 is now active with session: d304ebbb-e534-4d21-aaf8-8f822a610330","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963773037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963781516Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"session_id":"d304ebbb-e534-4d21-aaf8-8f822a610330"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963790505Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Activated sub_volume e6687d78-61a5-4863-8577-2666f39d54b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963798274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.963806164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.964099328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} 01/30 Write block 211 len 4096 data: 7 02/30 Write block 116 len 40960 data: 6 5 6 6 6 5 6 5 6 5 03/30 Read block 203 len 40960 04/30 Write block 247 len 12288 data: 7 7 7 05/30 Write block 147 len 28672 data: 7 4 4 4 5 5 8 06/30 Write block 78 len 16384 data: 5 4 5 4 07/30 Read block 322 len 40960 08/30 Read block 192 len 16384 09/30 Read block 461 len 24576 10/30 Read block 71 len 28672 11/30 Read block 318 len 8192 12/30 Read block 86 len 12288 13/30 Flush 14/30 Write block 338 len 36864 data: 6 7 8 6 5 5 5 6 6 15/30 Write block 414 len 4096 data: 5 16/30 Read block 476 len 24576 17/30 Read block 214 len 8192 18/30 Flush 19/30 Write block 404 len 4096 data: 9 20/30 Write block 324 len 32768 data: 10 8 9 8 8 6 6 5 21/30 Write block 381 len 20480 data: 8 7 7 6 6 22/30 Write block 101 len 40960 data: 5 5 6 7 6 5 6 6 5 7 23/30 Read block 471 len 12288 24/30 Write block 183 len 20480 data: 6 6 5 5 5 25/30 Write block 35 len 16384 data: 8 7 6 4 26/30 Flush 27/30 Write block 258 len 40960 data: 8 8 7 9 7 5 5 7 8 7 28/30 Write block 497 len 8192 data: 4 5 29/30 Read block 132 len 12288 30/30 Write block 441 len 36864 data: 7 9 8 11 9 11 9 10 8 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:21.972472229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:21.972645574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1387,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 58 62 62 98 103 103 F F F <--- 2 040-059 60 60 60 99 99 99 F F F 3 060-079 60 62 62 99 101 101 F F F <--- 4 080-099 57 62 62 97 101 101 F F F <--- 5 100-119 60 62 62 99 103 103 T F F <--- 6 120-139 58 62 62 98 101 101 T F F <--- 7 140-159 60 62 62 99 101 101 F F F <--- 8 160-179 50 50 50 84 84 84 F F F 9 180-199 56 62 62 96 103 103 F F F <--- 10 200-219 61 62 62 100 101 101 T F F <--- 11 220-239 56 56 56 93 93 93 F F F 12 240-259 61 62 62 100 101 101 F T T <--- 13 260-279 57 57 57 97 97 97 F T T <--- 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 61 62 62 100 103 103 F F F <--- 17 340-359 58 62 62 98 102 102 F F F <--- 18 360-379 58 58 58 98 98 98 F F F 19 380-399 58 62 62 98 103 103 F F F <--- 20 400-419 60 62 62 99 103 103 F F F <--- 21 420-439 56 56 56 94 94 94 F F F 22 440-459 61 61 61 100 100 100 F T T <--- 23 460-479 61 61 61 100 100 100 F F F 24 480-499 58 58 58 98 98 98 F T T <--- 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 58 58 58 98 98 98 F F F Max gen: 62, Max flush: 103 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 62 62 62 103 103 103 F F F 2 040-059 60 60 60 99 99 99 F F F 3 060-079 62 62 62 101 101 101 F F F 4 080-099 62 62 62 101 101 101 F F F 5 100-119 62 62 62 103 103 103 F F F 6 120-139 62 62 62 101 101 101 F F F 7 140-159 62 62 62 101 101 101 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 62 62 62 103 103 103 F F F 10 200-219 62 62 62 101 101 101 F F F 11 220-239 56 56 56 93 93 93 F F F 12 240-259 63 63 63 104 104 104 F F F 13 260-279 63 63 63 104 104 104 F F F 14 280-299 55 55 55 92 92 92 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 62 62 62 102 102 102 F F F 18 360-379 58 58 58 98 98 98 F F F 19 380-399 62 62 62 103 103 103 F F F 20 400-419 62 62 62 103 103 103 F F F 21 420-439 56 56 56 94 94 94 F F F 22 440-459 63 63 63 104 104 104 F F F 23 460-479 61 61 61 100 100 100 F F F 24 480-499 63 63 63 104 104 104 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 F F F 29 580-599 58 58 58 98 98 98 F F F Max gen: 63, Max flush: 104 Begin loop 32 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.236819576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245126872Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245408748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245438515Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245449194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245460743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Crucible 467e7643-d3d0-4d1d-8ce5-009763b46a4a has session id: 2f46d39b-da6b-4542-912e-cc38726d42e3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245770146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Upstairs opts: Upstairs UUID: 467e7643-d3d0-4d1d-8ce5-009763b46a4a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245794794Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Crucible stats registered with UUID: 467e7643-d3d0-4d1d-8ce5-009763b46a4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245804463Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:26.245813402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.24595624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245986207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.245997126Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246115746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246131545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"467e7643-d3d0-4d1d-8ce5-009763b46a4a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246149153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246157942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246167402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246177131Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.24618651Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246216827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246227116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:55495)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246252714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:62488)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246288091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:34639)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.2462987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"io task","client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246881849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.246900708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247010248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247275465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247284724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[0]R flush_numbers[0..12]: [99, 103, 99, 101, 101, 103, 101, 101, 84, 103, 101, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247321251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[0]R generation[0..12]: [60, 62, 60, 62, 62, 62, 62, 62, 50, 62, 62, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.24733201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247342999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[1]R flush_numbers[0..12]: [99, 103, 99, 101, 101, 103, 101, 101, 84, 103, 101, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247353408Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[1]R generation[0..12]: [60, 62, 60, 62, 62, 62, 62, 62, 50, 62, 62, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247365227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247375926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[2]R flush_numbers[0..12]: [99, 103, 99, 101, 101, 103, 101, 101, 84, 103, 101, 93]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247386275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[2]R generation[0..12]: [60, 62, 60, 62, 62, 62, 62, 62, 50, 62, 62, 56]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247396334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247406974Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247416303Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247425672Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Next flush: 105","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247434751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.24744395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"":"downstairs","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.24745341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247461819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"0","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247471158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"1","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247480627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"client":"2","":"downstairs","session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247490166Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247499266Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"467e7643-d3d0-4d1d-8ce5-009763b46a4a is now active with session: 2f46d39b-da6b-4542-912e-cc38726d42e3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247509105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247519004Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"session_id":"2f46d39b-da6b-4542-912e-cc38726d42e3"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247527893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Activated sub_volume 467e7643-d3d0-4d1d-8ce5-009763b46a4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247535762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247543682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.247878373Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} 01/30 Write block 126 len 28672 data: 6 7 7 7 8 9 8 02/30 Read block 567 len 36864 03/30 Write block 416 len 16384 data: 7 5 7 4 04/30 Write block 227 len 40960 data: 7 8 7 7 6 7 5 7 9 7 05/30 Read block 171 len 16384 06/30 Write block 285 len 28672 data: 5 5 5 4 3 3 4 07/30 Write block 343 len 20480 data: 6 6 7 7 6 08/30 Write block 429 len 28672 data: 7 8 6 6 5 4 4 09/30 Write block 190 len 12288 data: 4 5 5 10/30 Read block 390 len 4096 11/30 Flush 12/30 Write block 341 len 8192 data: 7 6 13/30 Write block 444 len 4096 data: 12 14/30 Write block 381 len 32768 data: 9 8 8 7 7 7 7 8 15/30 Write block 242 len 36864 data: 5 4 5 5 5 8 8 8 6 16/30 Read block 437 len 24576 17/30 Write block 63 len 36864 data: 5 4 4 4 5 5 5 5 5 18/30 Write block 591 len 4096 data: 11 19/30 Read block 377 len 20480 20/30 Write block 279 len 4096 data: 10 21/30 Write block 478 len 4096 data: 9 22/30 Read block 161 len 32768 23/30 Read block 11 len 24576 24/30 Read block 230 len 8192 25/30 Write block 37 len 16384 data: 7 5 4 3 26/30 Read block 167 len 12288 27/30 Write block 288 len 20480 data: 5 4 4 5 5 28/30 Write block 233 len 4096 data: 6 29/30 Write block 562 len 4096 data: 9 30/30 Read block 200 len 40960 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Write 1 Done Sent Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 2 29 Done 30 3 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) None Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:26.255426155Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:26.25560271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1395,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 62 62 62 103 103 103 T F T <--- 2 040-059 60 60 60 99 99 99 T F T <--- 3 060-079 62 62 62 101 101 101 T F T <--- 4 080-099 62 62 62 101 101 101 F F F 5 100-119 62 62 62 103 103 103 F F F 6 120-139 64 62 64 105 101 105 F T F <--- 7 140-159 62 62 62 101 101 101 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 64 62 64 105 103 105 F F F <--- 10 200-219 62 62 62 101 101 101 F F F 11 220-239 64 56 64 105 93 105 T F T <--- 12 240-259 63 63 63 104 104 104 T F T <--- 13 260-279 63 63 63 104 104 104 T F T <--- 14 280-299 64 55 64 105 92 105 T F T <--- 15 300-319 57 57 57 97 97 97 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 64 62 64 105 102 105 T F T <--- 18 360-379 58 58 58 98 98 98 F F F 19 380-399 62 62 62 103 103 103 T F T <--- 20 400-419 64 62 64 105 103 105 F T F <--- 21 420-439 64 56 64 105 94 105 F F F <--- 22 440-459 63 63 63 104 104 104 T F T <--- 23 460-479 61 61 61 100 100 100 T F T <--- 24 480-499 63 63 63 104 104 104 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 56 56 56 96 96 96 T F T <--- 29 580-599 58 58 58 98 98 98 T F T <--- Max gen: 64, Max flush: 105 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 65 65 65 106 106 106 F F F 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 62 62 62 101 101 101 F F F 5 100-119 62 62 62 103 103 103 F F F 6 120-139 64 64 64 105 105 105 F F F 7 140-159 62 62 62 101 101 101 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 64 64 64 105 105 105 F F F 10 200-219 62 62 62 101 101 101 F F F 11 220-239 65 65 65 106 106 106 F F F 12 240-259 65 65 65 106 106 106 F F F 13 260-279 65 65 65 106 106 106 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 57 57 57 97 97 97 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 65 65 65 106 106 106 F F F 18 360-379 58 58 58 98 98 98 F F F 19 380-399 65 65 65 106 106 106 F F F 20 400-419 64 64 64 105 105 105 F F F 21 420-439 64 64 64 105 105 105 F F F 22 440-459 65 65 65 106 106 106 F F F 23 460-479 65 65 65 106 106 106 F F F 24 480-499 63 63 63 104 104 104 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 65 65 65 106 106 106 F F F 29 580-599 65 65 65 106 106 106 F F F Max gen: 65, Max flush: 106 Begin loop 33 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.517742599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:30.524909954Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.524965379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532210808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532496013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532523141Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.53253159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532543069Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Crucible 5aeebe79-a222-4b0b-972c-acbd6fb044f7 has session id: 56224ead-77d7-47dc-a353-d3c7e1168037","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532861551Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Upstairs opts: Upstairs UUID: 5aeebe79-a222-4b0b-972c-acbd6fb044f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532885979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Crucible stats registered with UUID: 5aeebe79-a222-4b0b-972c-acbd6fb044f7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.532897698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:30.532906707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533047375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533063724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533073563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533256667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533273795Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"5aeebe79-a222-4b0b-972c-acbd6fb044f7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533296833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533305833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533315322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533324711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.5333341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533344229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"ds_connection connected from Ok(127.0.0.1:32941)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533385696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533396005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"ds_connection connected from Ok(127.0.0.1:55263)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533406034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"ds_connection connected from Ok(127.0.0.1:38310)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.533472378Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"io task","client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534050858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534078545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534118452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534388298Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534436254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[0]R flush_numbers[0..12]: [99, 106, 106, 106, 101, 103, 105, 101, 84, 105, 101, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534506308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[0]R generation[0..12]: [60, 65, 65, 65, 62, 62, 64, 62, 50, 64, 62, 65]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534517107Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534527976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[1]R flush_numbers[0..12]: [99, 106, 106, 106, 101, 103, 105, 101, 84, 105, 101, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534538275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[1]R generation[0..12]: [60, 65, 65, 65, 62, 62, 64, 62, 50, 64, 62, 65]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534548374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534559023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[2]R flush_numbers[0..12]: [99, 106, 106, 106, 101, 103, 105, 101, 84, 105, 101, 106]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534569353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[2]R generation[0..12]: [60, 65, 65, 65, 62, 62, 64, 62, 50, 64, 62, 65]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534579452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534592001Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.53460116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534610949Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Next flush: 107","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534620178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534629507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"":"downstairs","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534639226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534647846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"0","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534657325Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"1","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534666814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"client":"2","":"downstairs","session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534676453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534685422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"5aeebe79-a222-4b0b-972c-acbd6fb044f7 is now active with session: 56224ead-77d7-47dc-a353-d3c7e1168037","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534695212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534703811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"session_id":"56224ead-77d7-47dc-a353-d3c7e1168037"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.53471277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Activated sub_volume 5aeebe79-a222-4b0b-972c-acbd6fb044f7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534720659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.534728489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.535086178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} 01/30 Write block 135 len 20480 data: 7 8 8 7 9 02/30 Write block 432 len 28672 data: 7 6 5 5 5 6 6 03/30 Write block 103 len 20480 data: 7 8 7 6 7 04/30 Read block 117 len 40960 05/30 Flush 06/30 Write block 265 len 4096 data: 8 07/30 Read block 115 len 40960 08/30 Read block 248 len 36864 09/30 Read block 200 len 20480 10/30 Read block 539 len 32768 11/30 Read block 45 len 12288 12/30 Write block 31 len 32768 data: 5 5 6 8 9 8 8 6 13/30 Write block 303 len 8192 data: 12 10 14/30 Read block 423 len 28672 15/30 Write block 475 len 12288 data: 9 8 8 16/30 Write block 217 len 24576 data: 5 6 8 6 5 4 17/30 Write block 487 len 28672 data: 7 7 7 7 5 5 5 18/30 Read block 151 len 32768 19/30 Write block 269 len 28672 data: 6 6 6 8 8 8 7 20/30 Flush 21/30 Write block 154 len 8192 data: 9 8 22/30 Read block 270 len 16384 23/30 Write block 379 len 4096 data: 8 24/30 Read block 396 len 4096 25/30 Read block 247 len 40960 26/30 Write block 85 len 24576 data: 3 4 4 7 7 6 27/30 Write block 393 len 8192 data: 7 7 28/30 Read block 473 len 8192 29/30 Read block 37 len 28672 30/30 Write block 591 len 12288 data: 12 10 9 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:30.543610055Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:30.543771461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1403,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 65 66 66 106 108 108 F F F <--- 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 62 62 62 101 101 101 F T T <--- 5 100-119 62 66 66 103 107 107 T F F <--- 6 120-139 64 66 66 105 107 107 T F F <--- 7 140-159 62 62 62 101 101 101 F T T <--- 8 160-179 50 50 50 84 84 84 F F F 9 180-199 64 64 64 105 105 105 F F F 10 200-219 62 66 66 101 108 108 F F F <--- 11 220-239 65 66 66 106 108 108 F F F <--- 12 240-259 65 65 65 106 106 106 F F F 13 260-279 65 66 66 106 108 108 F F F <--- 14 280-299 65 65 65 106 106 106 F F F 15 300-319 57 66 66 97 108 108 F F F <--- 16 320-339 62 62 62 103 103 103 F F F 17 340-359 65 65 65 106 106 106 F F F 18 360-379 58 58 58 98 98 98 F T T <--- 19 380-399 65 65 65 106 106 106 F T T <--- 20 400-419 64 64 64 105 105 105 F F F 21 420-439 64 66 66 105 107 107 T F F <--- 22 440-459 65 65 65 106 106 106 F F F 23 460-479 65 66 66 106 108 108 F F F <--- 24 480-499 63 66 66 104 108 108 F F F <--- 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 65 65 65 106 106 106 F F F 29 580-599 65 65 65 106 106 106 F T T <--- Max gen: 66, Max flush: 108 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 99 99 99 F F F 1 020-039 66 66 66 108 108 108 F F F 2 040-059 65 65 65 106 106 106 F F F 3 060-079 65 65 65 106 106 106 F F F 4 080-099 67 67 67 109 109 109 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 66 66 66 107 107 107 F F F 7 140-159 67 67 67 109 109 109 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 64 64 64 105 105 105 F F F 10 200-219 66 66 66 108 108 108 F F F 11 220-239 66 66 66 108 108 108 F F F 12 240-259 65 65 65 106 106 106 F F F 13 260-279 66 66 66 108 108 108 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 66 66 66 108 108 108 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 65 65 65 106 106 106 F F F 18 360-379 67 67 67 109 109 109 F F F 19 380-399 67 67 67 109 109 109 F F F 20 400-419 64 64 64 105 105 105 F F F 21 420-439 66 66 66 107 107 107 F F F 22 440-459 65 65 65 106 106 106 F F F 23 460-479 66 66 66 108 108 108 F F F 24 480-499 66 66 66 108 108 108 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 58 58 98 98 98 F F F 28 560-579 65 65 65 106 106 106 F F F 29 580-599 67 67 67 109 109 109 F F F Max gen: 67, Max flush: 109 Begin loop 34 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.808516113Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:34.815774401Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.815839195Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.822974913Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823248179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823272807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823281346Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823292685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Crucible 45d8535d-193f-424e-98ac-48ae6bb9de71 has session id: dde9adce-400e-42a4-bfb9-521fc9d757e4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823634306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Upstairs opts: Upstairs UUID: 45d8535d-193f-424e-98ac-48ae6bb9de71, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823662863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Crucible stats registered with UUID: 45d8535d-193f-424e-98ac-48ae6bb9de71","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823673582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:34.823682671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823798871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.82381502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823828209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.823989965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824008973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"45d8535d-193f-424e-98ac-48ae6bb9de71 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824026381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824035171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.82404473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824054169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824063568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824073347Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824083007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:59479)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824132062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:53258)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824142801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:47437)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.82415278Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"io task","client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824791705Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824807303Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.824868788Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825107917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825117356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[0]R flush_numbers[0..12]: [99, 108, 106, 106, 109, 107, 107, 109, 84, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825180831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[0]R generation[0..12]: [60, 66, 65, 65, 67, 66, 66, 67, 50, 64, 66, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.82519176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825202529Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[1]R flush_numbers[0..12]: [99, 108, 106, 106, 109, 107, 107, 109, 84, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825212748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[1]R generation[0..12]: [60, 66, 65, 65, 67, 66, 66, 67, 50, 64, 66, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825222947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825233676Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[2]R flush_numbers[0..12]: [99, 108, 106, 106, 109, 107, 107, 109, 84, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825244145Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[2]R generation[0..12]: [60, 66, 65, 65, 67, 66, 66, 67, 50, 64, 66, 66]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825254245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825264814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825273933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825283302Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Next flush: 110","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825292361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.82530265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"":"downstairs","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825312149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825320739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"0","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825330548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"1","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825340267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"client":"2","":"downstairs","session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825350176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825359165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"45d8535d-193f-424e-98ac-48ae6bb9de71 is now active with session: dde9adce-400e-42a4-bfb9-521fc9d757e4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825368995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825377634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"session_id":"dde9adce-400e-42a4-bfb9-521fc9d757e4"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825386593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Activated sub_volume 45d8535d-193f-424e-98ac-48ae6bb9de71","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825394492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825402502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.825671608Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} 01/30 Write block 395 len 4096 data: 6 02/30 Read block 531 len 28672 03/30 Read block 561 len 28672 04/30 Write block 2 len 32768 data: 3 4 4 5 5 5 6 7 05/30 Read block 356 len 32768 06/30 Write block 38 len 12288 data: 7 5 4 07/30 Write block 185 len 4096 data: 6 08/30 Read block 441 len 4096 09/30 Write block 542 len 8192 data: 2 3 10/30 Read block 420 len 8192 11/30 Read block 537 len 20480 12/30 Read block 342 len 20480 13/30 Write block 235 len 36864 data: 10 8 8 7 6 6 6 6 5 14/30 Write block 590 len 16384 data: 11 13 11 10 15/30 Read block 159 len 8192 16/30 Read block 575 len 16384 17/30 Write block 396 len 36864 data: 4 5 9 7 8 10 10 9 10 18/30 Read block 480 len 20480 19/30 Read block 300 len 32768 20/30 Read block 173 len 28672 21/30 Read block 520 len 32768 22/30 Write block 450 len 28672 data: 10 8 5 6 6 6 6 23/30 Flush 24/30 Write block 361 len 8192 data: 10 9 25/30 Write block 414 len 28672 data: 6 6 8 6 8 5 5 26/30 Flush 27/30 Write block 354 len 40960 data: 6 5 5 5 5 6 6 11 10 11 28/30 Write block 77 len 40960 data: 5 6 5 6 5 3 3 3 4 5 29/30 Write block 189 len 16384 data: 4 5 6 6 30/30 Write block 348 len 12288 data: 6 7 9 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 10 Sent Sent Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 4 4 36 Done 2 26 26 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:34.834001052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:34.834167808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1411,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 68 68 99 110 110 F F F <--- 1 020-039 66 68 68 108 110 110 F F F <--- 2 040-059 65 68 68 106 110 110 F F F <--- 3 060-079 65 65 65 106 106 106 F T T <--- 4 080-099 67 67 67 109 109 109 F T T <--- 5 100-119 66 66 66 107 107 107 F F F 6 120-139 66 66 66 107 107 107 F F F 7 140-159 67 67 67 109 109 109 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 64 68 68 105 110 110 F T T <--- 10 200-219 66 66 66 108 108 108 F F F 11 220-239 66 68 68 108 110 110 F F F <--- 12 240-259 65 68 68 106 110 110 F F F <--- 13 260-279 66 66 66 108 108 108 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 66 66 66 108 108 108 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 65 65 65 106 106 106 F T T <--- 18 360-379 67 68 68 109 111 111 F T T <--- 19 380-399 67 68 68 109 110 110 T F F <--- 20 400-419 64 68 68 105 111 111 F F F <--- 21 420-439 66 68 68 107 111 111 F F F <--- 22 440-459 65 68 68 106 110 110 F F F <--- 23 460-479 66 66 66 108 108 108 F F F 24 480-499 66 66 66 108 108 108 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 58 68 68 98 110 110 F F F <--- 28 560-579 65 65 65 106 106 106 F F F 29 580-599 67 68 68 109 110 110 F F F <--- Max gen: 68, Max flush: 111 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 110 110 110 F F F 1 020-039 68 68 68 110 110 110 F F F 2 040-059 68 68 68 110 110 110 F F F 3 060-079 69 69 69 112 112 112 F F F 4 080-099 69 69 69 112 112 112 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 66 66 66 107 107 107 F F F 7 140-159 67 67 67 109 109 109 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 66 66 66 108 108 108 F F F 11 220-239 68 68 68 110 110 110 F F F 12 240-259 68 68 68 110 110 110 F F F 13 260-279 66 66 66 108 108 108 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 66 66 66 108 108 108 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 69 69 69 112 112 112 F F F 18 360-379 69 69 69 112 112 112 F F F 19 380-399 68 68 68 110 110 110 F F F 20 400-419 68 68 68 111 111 111 F F F 21 420-439 68 68 68 111 111 111 F F F 22 440-459 68 68 68 110 110 110 F F F 23 460-479 66 66 66 108 108 108 F F F 24 480-499 66 66 66 108 108 108 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 68 68 68 110 110 110 F F F 28 560-579 65 65 65 106 106 106 F F F 29 580-599 68 68 68 110 110 110 F F F Max gen: 69, Max flush: 112 Begin loop 35 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.097010497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:39.104220238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.104275194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111155594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111410582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11143513Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111443449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111454608Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Crucible aed88923-fc92-4de2-a22d-0087d5ac4f23 has session id: c494942d-d7f7-4d2c-8bbf-da0d0064ba29","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111756982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Upstairs opts: Upstairs UUID: aed88923-fc92-4de2-a22d-0087d5ac4f23, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11177979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Crucible stats registered with UUID: aed88923-fc92-4de2-a22d-0087d5ac4f23","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111789669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:39.111798558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111928987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111945545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.111955364Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112103392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11211858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"aed88923-fc92-4de2-a22d-0087d5ac4f23 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112151267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112162136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112171666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112181265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112190754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112200643Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112210492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"ds_connection connected from Ok(127.0.0.1:64078)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112409815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"ds_connection connected from Ok(127.0.0.1:52195)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112435543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"ds_connection connected from Ok(127.0.0.1:50752)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112446692Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"io task","client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11292815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.112955897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11303698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11326965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113306047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[0]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 107, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113393919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[0]R generation[0..12]: [68, 68, 68, 69, 69, 66, 66, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113407288Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113418237Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[1]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 107, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113428666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[1]R generation[0..12]: [68, 68, 68, 69, 69, 66, 66, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113438935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113449604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[2]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 107, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113459903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[2]R generation[0..12]: [68, 68, 68, 69, 69, 66, 66, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113469973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113480582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113489821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11349931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Next flush: 113","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113508439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113517738Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"":"downstairs","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113527558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113536107Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"0","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113546826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"1","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113556445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"client":"2","":"downstairs","session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113566214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113575213Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"aed88923-fc92-4de2-a22d-0087d5ac4f23 is now active with session: c494942d-d7f7-4d2c-8bbf-da0d0064ba29","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113584873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113593422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"session_id":"c494942d-d7f7-4d2c-8bbf-da0d0064ba29"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113602441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Activated sub_volume aed88923-fc92-4de2-a22d-0087d5ac4f23","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11361032Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.11361825Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.113925893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} 01/30 Write block 124 len 32768 data: 7 6 7 8 8 8 9 10 02/30 Write block 575 len 12288 data: 5 4 2 03/30 Write block 121 len 12288 data: 6 7 6 04/30 Flush 05/30 Write block 524 len 16384 data: 8 6 6 6 06/30 Read block 524 len 40960 07/30 Read block 387 len 8192 08/30 Write block 469 len 40960 data: 7 6 5 5 5 9 10 9 9 10 09/30 Read block 120 len 40960 10/30 Read block 520 len 28672 11/30 Write block 416 len 28672 data: 9 7 9 6 6 5 4 12/30 Flush 13/30 Write block 442 len 24576 data: 10 9 13 10 12 10 14/30 Write block 243 len 36864 data: 6 6 6 6 9 9 9 7 5 15/30 Write block 416 len 32768 data: 10 8 10 7 7 6 5 4 16/30 Read block 485 len 12288 17/30 Read block 552 len 4096 18/30 Read block 537 len 28672 19/30 Read block 52 len 28672 20/30 Read block 394 len 4096 21/30 Flush 22/30 Write block 267 len 16384 data: 8 7 7 7 23/30 Write block 555 len 28672 data: 7 5 5 5 6 6 8 24/30 Read block 554 len 4096 25/30 Read block 155 len 12288 26/30 Read block 106 len 24576 27/30 Read block 139 len 16384 28/30 Write block 304 len 8192 data: 11 9 29/30 Read block 393 len 32768 30/30 Write block 584 len 24576 data: 9 9 13 10 10 12 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:39.122496386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:39.122675331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1419,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 110 110 110 F F F 1 020-039 68 68 68 110 110 110 F F F 2 040-059 68 68 68 110 110 110 F F F 3 060-079 69 69 69 112 112 112 F F F 4 080-099 69 69 69 112 112 112 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 66 70 70 107 113 113 F F F <--- 7 140-159 67 67 67 109 109 109 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 66 66 66 108 108 108 F F F 11 220-239 68 68 68 110 110 110 F F F 12 240-259 68 70 70 110 115 115 F F F <--- 13 260-279 66 66 66 108 108 108 F T T <--- 14 280-299 65 65 65 106 106 106 F F F 15 300-319 66 66 66 108 108 108 F T T <--- 16 320-339 62 62 62 103 103 103 F F F 17 340-359 69 69 69 112 112 112 F F F 18 360-379 69 69 69 112 112 112 F F F 19 380-399 68 68 68 110 110 110 F F F 20 400-419 68 70 70 111 115 115 F F F <--- 21 420-439 68 70 70 111 115 115 F F F <--- 22 440-459 68 70 70 110 115 115 F F F <--- 23 460-479 66 70 70 108 114 114 F F F <--- 24 480-499 66 66 66 108 108 108 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 58 70 70 98 114 114 F F F <--- 27 540-559 68 68 68 110 110 110 F T T <--- 28 560-579 65 70 70 106 113 113 F T T <--- 29 580-599 68 68 68 110 110 110 F T T <--- Max gen: 70, Max flush: 115 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 110 110 110 F F F 1 020-039 68 68 68 110 110 110 F F F 2 040-059 68 68 68 110 110 110 F F F 3 060-079 69 69 69 112 112 112 F F F 4 080-099 69 69 69 112 112 112 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 F F F 7 140-159 67 67 67 109 109 109 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 66 66 66 108 108 108 F F F 11 220-239 68 68 68 110 110 110 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 71 71 71 116 116 116 F F F 14 280-299 65 65 65 106 106 106 F F F 15 300-319 71 71 71 116 116 116 F F F 16 320-339 62 62 62 103 103 103 F F F 17 340-359 69 69 69 112 112 112 F F F 18 360-379 69 69 69 112 112 112 F F F 19 380-399 68 68 68 110 110 110 F F F 20 400-419 70 70 70 115 115 115 F F F 21 420-439 70 70 70 115 115 115 F F F 22 440-459 70 70 70 115 115 115 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 66 66 66 108 108 108 F F F 25 500-519 58 58 58 98 98 98 F F F 26 520-539 70 70 70 114 114 114 F F F 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 71 71 71 116 116 116 F F F Max gen: 71, Max flush: 116 Begin loop 36 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.396816695Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.4052458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405538075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405563143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405571722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405583231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Crucible 9e379b0f-b77d-42dd-812b-0fa0f3de11bb has session id: a87cf188-c62c-45c4-b01c-2fb8f39e8923","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405877165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Upstairs opts: Upstairs UUID: 9e379b0f-b77d-42dd-812b-0fa0f3de11bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405911422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Crucible stats registered with UUID: 9e379b0f-b77d-42dd-812b-0fa0f3de11bb","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.405922231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:43.40593134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406068469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406086397Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406096426Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406266391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.40628309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"9e379b0f-b77d-42dd-812b-0fa0f3de11bb active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406317737Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406326946Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406336585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406346224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406355714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406365613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"ds_connection connected from Ok(127.0.0.1:57074)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406501341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"ds_connection connected from Ok(127.0.0.1:40835)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406525969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"ds_connection connected from Ok(127.0.0.1:51329)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.406537178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"io task","client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407208239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407228088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407244116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.40754961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407590986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[0]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 113, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407616154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[0]R generation[0..12]: [68, 68, 68, 69, 69, 66, 70, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407627283Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407638252Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[1]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 113, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407648731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[1]R generation[0..12]: [68, 68, 68, 69, 69, 66, 70, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.40765895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407669819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[2]R flush_numbers[0..12]: [110, 110, 110, 112, 112, 107, 113, 109, 84, 112, 108, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407682478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[2]R generation[0..12]: [68, 68, 68, 69, 69, 66, 70, 67, 50, 69, 66, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407692557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407703206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407712465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407721834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Next flush: 117","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407730864Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407740123Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"":"downstairs","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407749932Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407758561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"0","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.40776836Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"1","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.40777776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"client":"2","":"downstairs","session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407787439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407796398Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"9e379b0f-b77d-42dd-812b-0fa0f3de11bb is now active with session: a87cf188-c62c-45c4-b01c-2fb8f39e8923","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407806137Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407814596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"session_id":"a87cf188-c62c-45c4-b01c-2fb8f39e8923"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407824456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Activated sub_volume 9e379b0f-b77d-42dd-812b-0fa0f3de11bb","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407832275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.407840144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.408181654Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} 01/30 Write block 49 len 4096 data: 8 02/30 Read block 94 len 4096 03/30 Read block 579 len 32768 04/30 Flush 05/30 Write block 394 len 28672 data: 8 7 5 6 10 8 9 06/30 Write block 590 len 40960 data: 12 14 12 11 7 6 5 4 3 2 07/30 Write block 368 len 40960 data: 10 10 6 7 7 7 8 10 11 12 08/30 Flush 09/30 Write block 533 len 24576 data: 4 6 5 5 5 4 10/30 Read block 400 len 32768 11/30 Write block 209 len 36864 data: 5 5 8 7 6 5 6 5 6 12/30 Flush 13/30 Write block 487 len 4096 data: 8 14/30 Write block 92 len 12288 data: 4 4 3 15/30 Read block 467 len 16384 16/30 Write block 506 len 16384 data: 7 8 6 7 17/30 Read block 31 len 28672 18/30 Write block 368 len 28672 data: 11 11 7 8 8 8 9 19/30 Write block 420 len 36864 data: 8 7 6 5 4 4 4 4 7 20/30 Read block 173 len 40960 21/30 Read block 78 len 32768 22/30 Write block 343 len 32768 data: 7 7 8 8 7 7 8 10 23/30 Write block 293 len 28672 data: 7 8 10 11 9 9 11 24/30 Write block 392 len 4096 data: 11 25/30 Read block 270 len 4096 26/30 Write block 449 len 12288 data: 9 11 9 27/30 Write block 323 len 28672 data: 10 11 9 10 9 9 7 28/30 Write block 521 len 12288 data: 7 7 5 29/30 Write block 144 len 32768 data: 7 7 6 8 5 5 5 6 30/30 Write block 525 len 36864 data: 7 7 7 5 4 3 3 3 5 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 27 31 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:43.41638218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:43.416553075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1427,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 110 110 110 F F F 1 020-039 68 68 68 110 110 110 F F F 2 040-059 72 72 68 117 117 110 F F T <--- 3 060-079 69 69 69 112 112 112 F F F 4 080-099 69 69 69 112 112 112 T T F <--- 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 F F F 7 140-159 67 67 67 109 109 109 T T F <--- 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 72 72 66 119 119 108 F F F <--- 11 220-239 68 68 68 110 110 110 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 71 71 71 116 116 116 F F F 14 280-299 65 65 65 106 106 106 T T F <--- 15 300-319 71 71 71 116 116 116 F F F 16 320-339 62 62 62 103 103 103 T T F <--- 17 340-359 69 69 69 112 112 112 T T F <--- 18 360-379 72 72 69 118 118 112 T T F <--- 19 380-399 72 72 68 118 118 110 T T F <--- 20 400-419 72 72 70 118 118 115 F F F <--- 21 420-439 70 70 70 115 115 115 T T F <--- 22 440-459 70 70 70 115 115 115 T T F <--- 23 460-479 70 70 70 114 114 114 F F F 24 480-499 66 66 66 108 108 108 T T F <--- 25 500-519 58 58 58 98 98 98 T T F <--- 26 520-539 72 72 70 119 119 114 T T F <--- 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 72 72 71 118 118 116 F F F <--- Max gen: 72, Max flush: 119 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 110 110 110 F F F 1 020-039 68 68 68 110 110 110 F F F 2 040-059 72 72 72 117 117 117 F F F 3 060-079 69 69 69 112 112 112 F F F 4 080-099 73 73 73 120 120 120 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 F F F 7 140-159 73 73 73 120 120 120 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 72 72 72 119 119 119 F F F 11 220-239 68 68 68 110 110 110 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 71 71 71 116 116 116 F F F 14 280-299 73 73 73 120 120 120 F F F 15 300-319 71 71 71 116 116 116 F F F 16 320-339 73 73 73 120 120 120 F F F 17 340-359 73 73 73 120 120 120 F F F 18 360-379 73 73 73 120 120 120 F F F 19 380-399 73 73 73 120 120 120 F F F 20 400-419 72 72 72 118 118 118 F F F 21 420-439 73 73 73 120 120 120 F F F 22 440-459 73 73 73 120 120 120 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 73 73 73 120 120 120 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 73 73 73 120 120 120 F F F 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 72 72 72 118 118 118 F F F Max gen: 73, Max flush: 120 Begin loop 37 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.68081189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.688936572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689223527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689246575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689254835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689266244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Crucible b65a0b24-d505-4cdc-98a5-49b4780f61ae has session id: 32c2c9fc-dc07-40ce-a7ab-05979277ebac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689548939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Upstairs opts: Upstairs UUID: b65a0b24-d505-4cdc-98a5-49b4780f61ae, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689577836Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Crucible stats registered with UUID: b65a0b24-d505-4cdc-98a5-49b4780f61ae","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689589185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:47.689609214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689753341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689783678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689794328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689937675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689956223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"b65a0b24-d505-4cdc-98a5-49b4780f61ae active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.689984331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690001999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690011649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690021118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690030527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690044386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690054115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"ds_connection connected from Ok(127.0.0.1:59342)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690090152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"ds_connection connected from Ok(127.0.0.1:50995)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690100901Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"ds_connection connected from Ok(127.0.0.1:42734)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.69011425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"io task","client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690717197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.690855675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.69103261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691049098Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691116792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[0]R flush_numbers[0..12]: [110, 110, 117, 112, 120, 107, 113, 120, 84, 112, 119, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691326154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[0]R generation[0..12]: [68, 68, 72, 69, 73, 66, 70, 73, 50, 69, 72, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691337133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691347812Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[1]R flush_numbers[0..12]: [110, 110, 117, 112, 120, 107, 113, 120, 84, 112, 119, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691358271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[1]R generation[0..12]: [68, 68, 72, 69, 73, 66, 70, 73, 50, 69, 72, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.69136843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691379189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[2]R flush_numbers[0..12]: [110, 110, 117, 112, 120, 107, 113, 120, 84, 112, 119, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691389789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[2]R generation[0..12]: [68, 68, 72, 69, 73, 66, 70, 73, 50, 69, 72, 68]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691399938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691410547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691419886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691431335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691440454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691449683Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"":"downstairs","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691459183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691467672Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"0","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691477081Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"1","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.69148642Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"client":"2","":"downstairs","session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691495999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691504889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"b65a0b24-d505-4cdc-98a5-49b4780f61ae is now active with session: 32c2c9fc-dc07-40ce-a7ab-05979277ebac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691514538Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691523117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"session_id":"32c2c9fc-dc07-40ce-a7ab-05979277ebac"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691531976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Activated sub_volume b65a0b24-d505-4cdc-98a5-49b4780f61ae","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691540055Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691548065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.691867257Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} 01/30 Write block 1 len 36864 data: 2 4 5 5 6 6 6 7 8 02/30 Flush 03/30 Write block 49 len 16384 data: 9 7 7 7 04/30 Read block 453 len 40960 05/30 Read block 21 len 36864 06/30 Write block 263 len 8192 data: 6 6 07/30 Write block 68 len 32768 data: 6 6 6 6 6 5 4 5 08/30 Read block 508 len 28672 09/30 Read block 54 len 32768 10/30 Read block 248 len 28672 11/30 Write block 299 len 40960 data: 12 9 9 11 13 12 10 9 9 9 12/30 Read block 158 len 32768 13/30 Read block 416 len 40960 14/30 Read block 530 len 40960 15/30 Read block 287 len 40960 16/30 Read block 243 len 32768 17/30 Flush 18/30 Write block 149 len 36864 data: 6 6 7 6 9 10 9 7 8 19/30 Read block 502 len 4096 20/30 Write block 220 len 24576 data: 7 6 5 4 4 4 21/30 Read block 537 len 8192 22/30 Write block 28 len 24576 data: 4 2 2 6 6 7 23/30 Read block 283 len 40960 24/30 Read block 483 len 40960 25/30 Write block 67 len 36864 data: 6 7 7 7 7 7 6 5 6 26/30 Read block 187 len 36864 27/30 Write block 376 len 36864 data: 12 13 10 9 8 10 9 9 8 28/30 Write block 333 len 36864 data: 5 5 5 5 5 7 8 9 8 29/30 Read block 153 len 40960 30/30 Write block 282 len 16384 data: 6 6 6 6 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:47.702797665Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:47.702944712Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1435,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 68 121 121 110 F F T <--- 1 020-039 68 68 68 110 110 110 T T F <--- 2 040-059 74 74 72 122 122 117 F F F <--- 3 060-079 74 74 69 122 122 112 T T F <--- 4 080-099 73 73 73 120 120 120 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 F F F 7 140-159 73 73 73 120 120 120 T T F <--- 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 72 72 72 119 119 119 F F F 11 220-239 68 68 68 110 110 110 T T F <--- 12 240-259 70 70 70 115 115 115 F F F 13 260-279 74 74 71 122 122 116 F F F <--- 14 280-299 74 74 73 122 122 120 T T F <--- 15 300-319 74 74 71 122 122 116 F F F <--- 16 320-339 73 73 73 120 120 120 T T F <--- 17 340-359 73 73 73 120 120 120 T T F <--- 18 360-379 73 73 73 120 120 120 T T F <--- 19 380-399 73 73 73 120 120 120 T T F <--- 20 400-419 72 72 72 118 118 118 F F F 21 420-439 73 73 73 120 120 120 F F F 22 440-459 73 73 73 120 120 120 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 73 73 73 120 120 120 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 73 73 73 120 120 120 F F F 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 72 72 72 118 118 118 F F F Max gen: 74, Max flush: 122 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 74 74 74 122 122 122 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 73 73 73 120 120 120 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 F F F 7 140-159 75 75 75 123 123 123 F F F 8 160-179 50 50 50 84 84 84 F F F 9 180-199 69 69 69 112 112 112 F F F 10 200-219 72 72 72 119 119 119 F F F 11 220-239 75 75 75 123 123 123 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 74 74 74 122 122 122 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 74 74 74 122 122 122 F F F 16 320-339 75 75 75 123 123 123 F F F 17 340-359 75 75 75 123 123 123 F F F 18 360-379 75 75 75 123 123 123 F F F 19 380-399 75 75 75 123 123 123 F F F 20 400-419 72 72 72 118 118 118 F F F 21 420-439 73 73 73 120 120 120 F F F 22 440-459 73 73 73 120 120 120 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 73 73 73 120 120 120 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 73 73 73 120 120 120 F F F 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 72 72 72 118 118 118 F F F Max gen: 75, Max flush: 123 Begin loop 38 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.957484983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.965688569Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.965971724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966005961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96601492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966026739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Crucible 4aa719f6-2212-4d46-ac1d-9043d8e7049d has session id: 9173ec5d-e452-4a6d-9a2b-2859477545c3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966370779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Upstairs opts: Upstairs UUID: 4aa719f6-2212-4d46-ac1d-9043d8e7049d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966399077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Crucible stats registered with UUID: 4aa719f6-2212-4d46-ac1d-9043d8e7049d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966409556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:51.966418385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966559683Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966575861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96658576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966737977Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966753696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"4aa719f6-2212-4d46-ac1d-9043d8e7049d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966771664Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966782603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966792152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966801442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966810881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96682081Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966830629Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:54026)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966881665Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:62430)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.966918471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47564)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96693862Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"io task","client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96750832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967521589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967539157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96785761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967870589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[0]R flush_numbers[0..12]: [121, 123, 122, 123, 120, 107, 113, 123, 84, 112, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967885177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[0]R generation[0..12]: [74, 75, 74, 75, 73, 66, 70, 75, 50, 69, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967897966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967908845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[1]R flush_numbers[0..12]: [121, 123, 122, 123, 120, 107, 113, 123, 84, 112, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967919304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[1]R generation[0..12]: [74, 75, 74, 75, 73, 66, 70, 75, 50, 69, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967929493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967940072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[2]R flush_numbers[0..12]: [121, 123, 122, 123, 120, 107, 113, 123, 84, 112, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967950342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[2]R generation[0..12]: [74, 75, 74, 75, 73, 66, 70, 75, 50, 69, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967960541Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96797122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967980769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967990308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Next flush: 124","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.967999467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968008906Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"":"downstairs","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968018886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968027635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"0","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968037224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"1","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968047693Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"client":"2","":"downstairs","session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968057562Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968066611Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"4aa719f6-2212-4d46-ac1d-9043d8e7049d is now active with session: 9173ec5d-e452-4a6d-9a2b-2859477545c3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968076381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96808483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"session_id":"9173ec5d-e452-4a6d-9a2b-2859477545c3"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968093719Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Activated sub_volume 4aa719f6-2212-4d46-ac1d-9043d8e7049d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968101548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.968109398Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.96842593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} 01/30 Flush 02/30 Write block 41 len 32768 data: 4 3 3 5 5 6 6 9 03/30 Read block 373 len 12288 04/30 Read block 21 len 32768 05/30 Write block 597 len 8192 data: 5 4 06/30 Write block 168 len 40960 data: 4 4 5 5 5 6 4 3 4 4 07/30 Write block 389 len 32768 data: 7 6 8 12 8 9 8 6 08/30 Write block 380 len 28672 data: 9 11 10 10 9 8 8 09/30 Read block 457 len 40960 10/30 Read block 50 len 36864 11/30 Flush 12/30 Write block 484 len 40960 data: 8 6 8 9 8 8 8 6 6 6 13/30 Write block 310 len 4096 data: 9 14/30 Write block 588 len 20480 data: 11 13 13 15 13 15/30 Read block 211 len 4096 16/30 Write block 406 len 28672 data: 6 6 4 4 5 4 4 17/30 Write block 525 len 32768 data: 8 8 8 6 5 4 4 4 18/30 Write block 159 len 8192 data: 7 6 19/30 Read block 589 len 28672 20/30 Read block 317 len 32768 21/30 Write block 80 len 40960 data: 7 6 4 4 4 5 6 5 8 8 22/30 Write block 423 len 36864 data: 6 5 5 5 5 8 8 9 7 23/30 Flush 24/30 Write block 189 len 8192 data: 5 6 25/30 Read block 548 len 16384 26/30 Write block 391 len 20480 data: 9 13 9 10 9 27/30 Write block 122 len 28672 data: 8 7 8 7 8 9 9 28/30 Read block 507 len 36864 29/30 Read block 383 len 24576 30/30 Read block 118 len 32768 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 1 27 Done 29 3 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1000)) Some(JobId(1022)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:51.977282159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:51.977452914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1443,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 76 74 76 125 122 125 F T F <--- 3 060-079 75 75 75 123 123 123 F F F 4 080-099 76 73 76 126 120 126 F F F <--- 5 100-119 66 66 66 107 107 107 F F F 6 120-139 70 70 70 113 113 113 T F T <--- 7 140-159 76 75 76 126 123 126 F F F <--- 8 160-179 76 50 76 126 84 126 F F F <--- 9 180-199 69 69 69 112 112 112 T F T <--- 10 200-219 72 72 72 119 119 119 F F F 11 220-239 75 75 75 123 123 123 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 74 74 74 122 122 122 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 76 74 76 126 122 126 F F F <--- 16 320-339 75 75 75 123 123 123 F F F 17 340-359 75 75 75 123 123 123 F F F 18 360-379 75 75 75 123 123 123 F F F 19 380-399 76 75 76 125 123 125 T F T <--- 20 400-419 76 72 76 126 118 126 F F F <--- 21 420-439 76 73 76 126 120 126 F F F <--- 22 440-459 73 73 73 120 120 120 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 76 73 76 126 120 126 F F F <--- 25 500-519 73 73 73 120 120 120 F F F 26 520-539 76 73 76 126 120 126 F F F <--- 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 76 72 76 126 118 126 F F F <--- Max gen: 76, Max flush: 126 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 76 76 76 125 125 125 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 76 76 76 126 126 126 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 77 77 77 127 127 127 F F F 7 140-159 76 76 76 126 126 126 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 72 72 72 119 119 119 F F F 11 220-239 75 75 75 123 123 123 F F F 12 240-259 70 70 70 115 115 115 F F F 13 260-279 74 74 74 122 122 122 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 76 76 76 126 126 126 F F F 16 320-339 75 75 75 123 123 123 F F F 17 340-359 75 75 75 123 123 123 F F F 18 360-379 75 75 75 123 123 123 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 76 76 76 126 126 126 F F F 21 420-439 76 76 76 126 126 126 F F F 22 440-459 73 73 73 120 120 120 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 76 76 76 126 126 126 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 76 76 76 126 126 126 F F F 27 540-559 71 71 71 116 116 116 F F F 28 560-579 71 71 71 116 116 116 F F F 29 580-599 76 76 76 126 126 126 F F F Max gen: 77, Max flush: 127 Begin loop 39 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.228184917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.236405481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.236692266Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.236716164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.236724533Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.236735702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Crucible 0dcc3776-004e-45e3-b7f1-52a62b1b6776 has session id: d3e844db-b03a-4703-a1d6-7b39e978fe2e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237058934Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Upstairs opts: Upstairs UUID: 0dcc3776-004e-45e3-b7f1-52a62b1b6776, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237093691Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Crucible stats registered with UUID: 0dcc3776-004e-45e3-b7f1-52a62b1b6776","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.23710487Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:56.237114159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237269296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237299443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237310032Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237443961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237459629Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"0dcc3776-004e-45e3-b7f1-52a62b1b6776 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237477228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237485967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237495376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237504825Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237514194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237524054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237536033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:65328)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237606866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:43292)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237638684Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39270)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.237649863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"io task","client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238295756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238316485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238330153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238627277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238639286Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[0]R flush_numbers[0..12]: [121, 123, 125, 123, 126, 107, 127, 126, 126, 127, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238678973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[0]R generation[0..12]: [74, 75, 76, 75, 76, 66, 77, 76, 76, 77, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238689952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238700731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[1]R flush_numbers[0..12]: [121, 123, 125, 123, 126, 107, 127, 126, 126, 127, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.23871121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[1]R generation[0..12]: [74, 75, 76, 75, 76, 66, 77, 76, 76, 77, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238721619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238734038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[2]R flush_numbers[0..12]: [121, 123, 125, 123, 126, 107, 127, 126, 126, 127, 119, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238744617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[2]R generation[0..12]: [74, 75, 76, 75, 76, 66, 77, 76, 76, 77, 72, 75]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238754786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238765675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238774895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238784404Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Next flush: 128","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238793603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238802702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"":"downstairs","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238812291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238820731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"0","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.2388302Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"1","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238839969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"client":"2","":"downstairs","session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238849878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238858767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"0dcc3776-004e-45e3-b7f1-52a62b1b6776 is now active with session: d3e844db-b03a-4703-a1d6-7b39e978fe2e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238868456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238876986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"session_id":"d3e844db-b03a-4703-a1d6-7b39e978fe2e"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238886945Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Activated sub_volume 0dcc3776-004e-45e3-b7f1-52a62b1b6776","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238894794Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.238902663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.239234535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} 01/30 Write block 215 len 24576 data: 7 6 7 7 9 8 02/30 Write block 258 len 36864 data: 9 9 8 10 8 7 7 9 9 03/30 Read block 115 len 40960 04/30 Read block 308 len 20480 05/30 Read block 380 len 16384 06/30 Write block 153 len 20480 data: 10 11 10 8 9 07/30 Write block 439 len 12288 data: 6 7 8 08/30 Read block 156 len 40960 09/30 Read block 336 len 28672 10/30 Flush 11/30 Flush 12/30 Write block 426 len 36864 data: 6 6 9 9 10 8 8 7 6 13/30 Write block 246 len 16384 data: 7 10 10 10 14/30 Write block 532 len 40960 data: 5 6 7 6 6 6 5 4 3 2 15/30 Write block 208 len 12288 data: 5 6 6 16/30 Read block 63 len 12288 17/30 Write block 50 len 4096 data: 8 18/30 Write block 315 len 32768 data: 7 7 7 7 9 9 8 10 19/30 Read block 290 len 32768 20/30 Read block 538 len 40960 21/30 Write block 264 len 32768 data: 8 10 10 9 8 8 8 7 22/30 Read block 222 len 40960 23/30 Flush 24/30 Write block 566 len 16384 data: 4 3 3 5 25/30 Read block 303 len 4096 26/30 Write block 534 len 36864 data: 8 7 7 7 6 5 4 3 3 27/30 Read block 585 len 16384 28/30 Write block 373 len 12288 data: 9 10 11 29/30 Read block 383 len 24576 30/30 Write block 325 len 32768 data: 10 11 10 10 8 7 6 4 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 4 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Read 10 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 29 29 4 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:30:56.248798831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:30:56.248977146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1451,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 78 78 76 130 130 125 F F F <--- 3 060-079 75 75 75 123 123 123 F F F 4 080-099 76 76 76 126 126 126 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 77 77 77 127 127 127 F F F 7 140-159 78 78 76 128 128 126 F F F <--- 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 78 78 72 130 130 119 F F T <--- 11 220-239 78 78 75 128 128 123 F F T <--- 12 240-259 78 78 70 130 130 115 F F T <--- 13 260-279 78 78 74 130 130 122 F F T <--- 14 280-299 75 75 75 123 123 123 F F F 15 300-319 78 78 76 130 130 126 F F F <--- 16 320-339 78 78 75 130 130 123 T T F <--- 17 340-359 75 75 75 123 123 123 F F F 18 360-379 75 75 75 123 123 123 T T F <--- 19 380-399 77 77 77 127 127 127 F F F 20 400-419 76 76 76 126 126 126 F F F 21 420-439 78 78 76 130 130 126 F F F <--- 22 440-459 78 78 73 128 128 120 F F F <--- 23 460-479 70 70 70 114 114 114 F F F 24 480-499 76 76 76 126 126 126 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 78 78 76 130 130 126 T T F <--- 27 540-559 78 78 71 130 130 116 T T F <--- 28 560-579 71 71 71 116 116 116 T T F <--- 29 580-599 76 76 76 126 126 126 F F F Max gen: 78, Max flush: 130 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 78 78 78 130 130 130 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 76 76 76 126 126 126 F F F 5 100-119 66 66 66 107 107 107 F F F 6 120-139 77 77 77 127 127 127 F F F 7 140-159 78 78 78 128 128 128 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 78 78 78 130 130 130 F F F 11 220-239 78 78 78 128 128 128 F F F 12 240-259 78 78 78 130 130 130 F F F 13 260-279 78 78 78 130 130 130 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 78 78 78 130 130 130 F F F 16 320-339 79 79 79 131 131 131 F F F 17 340-359 75 75 75 123 123 123 F F F 18 360-379 79 79 79 131 131 131 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 76 76 76 126 126 126 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 76 76 76 126 126 126 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 79 79 79 131 131 131 F F F 28 560-579 79 79 79 131 131 131 F F F 29 580-599 76 76 76 126 126 126 F F F Max gen: 79, Max flush: 131 Begin loop 40 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.503809891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.511962351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512248236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512274924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512283473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512294802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Crucible 661637e9-13d6-44ee-8f41-a03055a5d6d1 has session id: 963fecc4-51c2-4dce-9d4c-99c1044eb924","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512585546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Upstairs opts: Upstairs UUID: 661637e9-13d6-44ee-8f41-a03055a5d6d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512607305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Crucible stats registered with UUID: 661637e9-13d6-44ee-8f41-a03055a5d6d1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512616844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:00.512625773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.51277067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512787089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512797098Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512944285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512956484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"661637e9-13d6-44ee-8f41-a03055a5d6d1 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512979762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.512988961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.51299861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.51300811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513017509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513027248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513037097Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"ds_connection connected from Ok(127.0.0.1:41658)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513135219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"ds_connection connected from Ok(127.0.0.1:33854)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513246599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"ds_connection connected from Ok(127.0.0.1:34034)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513291035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"io task","client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513727587Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513742756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.513826498Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514023491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.51404191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[0]R flush_numbers[0..12]: [121, 123, 130, 123, 126, 107, 127, 128, 126, 127, 130, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514095925Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[0]R generation[0..12]: [74, 75, 78, 75, 76, 66, 77, 78, 76, 77, 78, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514107084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514118113Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[1]R flush_numbers[0..12]: [121, 123, 130, 123, 126, 107, 127, 128, 126, 127, 130, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514128722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[1]R generation[0..12]: [74, 75, 78, 75, 76, 66, 77, 78, 76, 77, 78, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514138951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.51414967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[2]R flush_numbers[0..12]: [121, 123, 130, 123, 126, 107, 127, 128, 126, 127, 130, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514160169Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[2]R generation[0..12]: [74, 75, 78, 75, 76, 66, 77, 78, 76, 77, 78, 78]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514170348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514181077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514190507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514201526Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Next flush: 132","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514210745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514220094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"":"downstairs","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514229593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514238372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"0","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514247922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"1","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514257271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"client":"2","":"downstairs","session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514276269Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"661637e9-13d6-44ee-8f41-a03055a5d6d1 is now active with session: 963fecc4-51c2-4dce-9d4c-99c1044eb924","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514286108Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514294788Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"session_id":"963fecc4-51c2-4dce-9d4c-99c1044eb924"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514303857Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Activated sub_volume 661637e9-13d6-44ee-8f41-a03055a5d6d1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514311866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514325355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.514591402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} 01/30 Write block 354 len 4096 data: 7 02/30 Read block 37 len 20480 03/30 Read block 302 len 8192 04/30 Read block 426 len 40960 05/30 Flush 06/30 Write block 95 len 32768 data: 2 2 3 4 5 5 6 6 07/30 Write block 143 len 4096 data: 5 08/30 Write block 122 len 8192 data: 9 8 09/30 Write block 366 len 12288 data: 9 10 12 10/30 Write block 402 len 40960 data: 11 10 11 8 7 7 5 5 6 5 11/30 Read block 174 len 12288 12/30 Write block 237 len 40960 data: 9 8 7 7 7 7 7 7 7 8 13/30 Write block 226 len 24576 data: 6 8 9 8 8 7 14/30 Read block 559 len 40960 15/30 Write block 556 len 40960 data: 6 6 6 7 7 9 10 7 6 5 16/30 Read block 579 len 20480 17/30 Read block 369 len 8192 18/30 Read block 102 len 4096 19/30 Read block 431 len 12288 20/30 Write block 15 len 20480 data: 7 6 6 5 6 21/30 Read block 587 len 16384 22/30 Write block 236 len 20480 data: 9 10 9 8 8 23/30 Read block 531 len 16384 24/30 Read block 22 len 20480 25/30 Write block 43 len 28672 data: 4 6 6 7 7 10 10 26/30 Read block 521 len 36864 27/30 Write block 493 len 12288 data: 7 4 4 28/30 Read block 441 len 20480 29/30 Read block 400 len 16384 30/30 Read block 500 len 8192 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Read 5 Done Done Sent false 1002 Acked Read 2 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 8 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Read 4 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 30 31 Done 29 30 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1004)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:00.522589815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:00.52276616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1459,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 121 121 121 T T F <--- 1 020-039 75 75 75 123 123 123 F F F 2 040-059 78 78 78 130 130 130 T T F <--- 3 060-079 75 75 75 123 123 123 F F F 4 080-099 76 76 76 126 126 126 T T F <--- 5 100-119 66 66 66 107 107 107 T T F <--- 6 120-139 77 77 77 127 127 127 T T F <--- 7 140-159 78 78 78 128 128 128 T T F <--- 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 78 78 78 130 130 130 F F F 11 220-239 78 78 78 128 128 128 T T F <--- 12 240-259 78 78 78 130 130 130 T T F <--- 13 260-279 78 78 78 130 130 130 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 78 78 78 130 130 130 F F F 16 320-339 79 79 79 131 131 131 F F F 17 340-359 80 80 75 132 132 123 F F F <--- 18 360-379 79 79 79 131 131 131 T T F <--- 19 380-399 77 77 77 127 127 127 F F F 20 400-419 76 76 76 126 126 126 T T F <--- 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 76 76 76 126 126 126 T T F <--- 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 79 79 79 131 131 131 T T F <--- 28 560-579 79 79 79 131 131 131 T T F <--- 29 580-599 76 76 76 126 126 126 F F F Max gen: 80, Max flush: 132 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 81 81 81 133 133 133 F F F 1 020-039 75 75 75 123 123 123 F F F 2 040-059 81 81 81 133 133 133 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 81 81 81 133 133 133 F F F 5 100-119 81 81 81 133 133 133 F F F 6 120-139 81 81 81 133 133 133 F F F 7 140-159 81 81 81 133 133 133 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 78 78 78 130 130 130 F F F 11 220-239 81 81 81 133 133 133 F F F 12 240-259 81 81 81 133 133 133 F F F 13 260-279 78 78 78 130 130 130 F F F 14 280-299 75 75 75 123 123 123 F F F 15 300-319 78 78 78 130 130 130 F F F 16 320-339 79 79 79 131 131 131 F F F 17 340-359 80 80 80 132 132 132 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 81 81 81 133 133 133 F F F 28 560-579 81 81 81 133 133 133 F F F 29 580-599 76 76 76 126 126 126 F F F Max gen: 81, Max flush: 133 Begin loop 41 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.782046508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790268621Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790559096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790593033Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790602012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790613611Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Crucible 3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b has session id: 1d2cba08-8494-460c-b88e-17a1ebf822d0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790897697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Upstairs opts: Upstairs UUID: 3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790927624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Crucible stats registered with UUID: 3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.790941713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:04.790950572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791100819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791131616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791142275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791271174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791283203Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.79131907Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791330519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791340218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791349727Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791358926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791368595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791378405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:35836)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791416301Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:40791)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791440929Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:44595)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.791452178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"io task","client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792082563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792139038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792162306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792442602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792486518Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[0]R flush_numbers[0..12]: [133, 123, 133, 123, 133, 133, 133, 133, 126, 127, 130, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792544753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[0]R generation[0..12]: [81, 75, 81, 75, 81, 81, 81, 81, 76, 77, 78, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792557712Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792568661Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[1]R flush_numbers[0..12]: [133, 123, 133, 123, 133, 133, 133, 133, 126, 127, 130, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.79257908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[1]R generation[0..12]: [81, 75, 81, 75, 81, 81, 81, 81, 76, 77, 78, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792589509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792600328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[2]R flush_numbers[0..12]: [133, 123, 133, 123, 133, 133, 133, 133, 126, 127, 130, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792610597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[2]R generation[0..12]: [81, 75, 81, 75, 81, 81, 81, 81, 76, 77, 78, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792620596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792631205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792640385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792649844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792659013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792668112Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"":"downstairs","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792677681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792686341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"0","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.79269594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"1","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792706699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"client":"2","":"downstairs","session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792716428Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792725337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b is now active with session: 1d2cba08-8494-460c-b88e-17a1ebf822d0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792735016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792743446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"session_id":"1d2cba08-8494-460c-b88e-17a1ebf822d0"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792752185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Activated sub_volume 3bdef146-fe5a-4bbf-b9b4-dcbed3b2d87b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792759924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.792767904Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.793108414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} 01/30 Write block 294 len 8192 data: 9 11 02/30 Flush 03/30 Write block 319 len 20480 data: 10 10 9 11 11 04/30 Write block 474 len 8192 data: 10 11 05/30 Read block 405 len 4096 06/30 Read block 333 len 24576 07/30 Flush 08/30 Write block 353 len 28672 data: 5 8 6 6 6 6 7 09/30 Flush 10/30 Write block 22 len 36864 data: 2 2 4 4 4 4 5 3 3 11/30 Read block 74 len 12288 12/30 Read block 484 len 28672 13/30 Read block 124 len 40960 14/30 Read block 45 len 40960 15/30 Write block 463 len 20480 data: 7 6 7 6 5 16/30 Read block 61 len 32768 17/30 Write block 587 len 20480 data: 11 12 14 14 16 18/30 Read block 21 len 16384 19/30 Write block 210 len 28672 data: 7 9 8 7 6 8 7 20/30 Read block 222 len 32768 21/30 Read block 513 len 40960 22/30 Read block 563 len 40960 23/30 Read block 333 len 36864 24/30 Write block 565 len 32768 data: 6 5 4 4 6 3 5 5 25/30 Read block 221 len 36864 26/30 Read block 429 len 12288 27/30 Read block 153 len 12288 28/30 Read block 255 len 8192 29/30 Write block 260 len 28672 data: 9 11 9 8 9 11 11 30/30 Read block 41 len 8192 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 5 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) None Some(JobId(1008)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:04.802273875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:04.80244751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1467,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 81 81 81 133 133 133 F F F 1 020-039 75 75 75 123 123 123 T F T <--- 2 040-059 81 81 81 133 133 133 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 81 81 81 133 133 133 F F F 5 100-119 81 81 81 133 133 133 F F F 6 120-139 81 81 81 133 133 133 F F F 7 140-159 81 81 81 133 133 133 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 78 78 78 130 130 130 T F T <--- 11 220-239 81 81 81 133 133 133 F F F 12 240-259 81 81 81 133 133 133 F F F 13 260-279 78 78 78 130 130 130 T F T <--- 14 280-299 82 75 82 134 123 134 F T F <--- 15 300-319 82 78 82 135 130 135 F F F <--- 16 320-339 82 79 82 135 131 135 F F F <--- 17 340-359 82 80 82 136 132 136 F F F <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 82 70 82 135 114 135 T F T <--- 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 81 81 81 133 133 133 F F F 28 560-579 81 81 81 133 133 133 T F T <--- 29 580-599 76 76 76 126 126 126 T F T <--- Max gen: 82, Max flush: 136 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 81 81 81 133 133 133 F F F 1 020-039 83 83 83 137 137 137 F F F 2 040-059 81 81 81 133 133 133 F F F 3 060-079 75 75 75 123 123 123 F F F 4 080-099 81 81 81 133 133 133 F F F 5 100-119 81 81 81 133 133 133 F F F 6 120-139 81 81 81 133 133 133 F F F 7 140-159 81 81 81 133 133 133 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 77 77 77 127 127 127 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 81 81 81 133 133 133 F F F 12 240-259 81 81 81 133 133 133 F F F 13 260-279 83 83 83 137 137 137 F F F 14 280-299 82 82 82 134 134 134 F F F 15 300-319 82 82 82 135 135 135 F F F 16 320-339 82 82 82 135 135 135 F F F 17 340-359 82 82 82 136 136 136 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 81 81 81 133 133 133 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 83 83 83 137 137 137 F F F Max gen: 83, Max flush: 137 Begin loop 42 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.052580234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.060936746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061258208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061283436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061291985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061303224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Crucible 8cd9f5c7-8181-4589-bbeb-933857e76e4a has session id: 6a6092f7-8eb3-458f-a579-7e7f67021eda","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061601449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Upstairs opts: Upstairs UUID: 8cd9f5c7-8181-4589-bbeb-933857e76e4a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061625966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Crucible stats registered with UUID: 8cd9f5c7-8181-4589-bbeb-933857e76e4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061635876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:09.061644685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061789022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.06181877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061829579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061937409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061954168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"8cd9f5c7-8181-4589-bbeb-933857e76e4a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061978036Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.061987255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062006663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062016242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062025462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062035131Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.06204479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"ds_connection connected from Ok(127.0.0.1:59161)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062077277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"ds_connection connected from Ok(127.0.0.1:58800)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062088176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"ds_connection connected from Ok(127.0.0.1:52539)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062098395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"io task","client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062768047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.062792525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.06284747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063069861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063088969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[0]R flush_numbers[0..12]: [133, 137, 133, 123, 133, 133, 133, 133, 126, 127, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063160503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[0]R generation[0..12]: [81, 83, 81, 75, 81, 81, 81, 81, 76, 77, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063171402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063182221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[1]R flush_numbers[0..12]: [133, 137, 133, 123, 133, 133, 133, 133, 126, 127, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.06319278Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[1]R generation[0..12]: [81, 83, 81, 75, 81, 81, 81, 81, 76, 77, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063203099Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063213878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[2]R flush_numbers[0..12]: [133, 137, 133, 123, 133, 133, 133, 133, 126, 127, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063226427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[2]R generation[0..12]: [81, 83, 81, 75, 81, 81, 81, 81, 76, 77, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063236546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063247245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063256524Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063266083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Next flush: 138","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063275223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063284252Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"":"downstairs","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063293721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.06330238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"0","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063311999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"1","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063321379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"client":"2","":"downstairs","session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063331038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063339987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"8cd9f5c7-8181-4589-bbeb-933857e76e4a is now active with session: 6a6092f7-8eb3-458f-a579-7e7f67021eda","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063349656Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063358245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"session_id":"6a6092f7-8eb3-458f-a579-7e7f67021eda"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063367275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Activated sub_volume 8cd9f5c7-8181-4589-bbeb-933857e76e4a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063376624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063384443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.063706135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} 01/30 Write block 240 len 4096 data: 9 02/30 Read block 291 len 28672 03/30 Read block 477 len 36864 04/30 Read block 262 len 24576 05/30 Read block 77 len 8192 06/30 Read block 383 len 28672 07/30 Write block 332 len 28672 data: 5 6 6 6 6 6 8 08/30 Read block 121 len 24576 09/30 Read block 438 len 16384 10/30 Write block 242 len 8192 data: 8 8 11/30 Read block 597 len 12288 12/30 Read block 253 len 20480 13/30 Write block 86 len 40960 data: 7 6 9 9 7 4 5 5 4 3 14/30 Read block 68 len 32768 15/30 Write block 190 len 32768 data: 7 7 7 4 4 6 5 5 16/30 Read block 234 len 24576 17/30 Flush 18/30 Write block 590 len 12288 data: 15 17 14 19/30 Write block 79 len 16384 data: 6 8 7 5 20/30 Read block 195 len 4096 21/30 Read block 502 len 4096 22/30 Read block 504 len 24576 23/30 Read block 114 len 4096 24/30 Read block 142 len 28672 25/30 Read block 526 len 4096 26/30 Write block 545 len 32768 data: 3 2 2 3 3 3 3 5 27/30 Write block 20 len 32768 data: 4 3 3 3 5 5 5 5 28/30 Read block 449 len 8192 29/30 Read block 171 len 20480 30/30 Read block 348 len 32768 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 7 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 24 25 Done 30 29 6 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:09.072527497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:09.072700342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1475,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 81 81 81 133 133 133 F F F 1 020-039 83 83 83 137 137 137 T T F <--- 2 040-059 81 81 81 133 133 133 F F F 3 060-079 75 75 75 123 123 123 T T F <--- 4 080-099 84 84 81 138 138 133 T T F <--- 5 100-119 81 81 81 133 133 133 F F F 6 120-139 81 81 81 133 133 133 F F F 7 140-159 81 81 81 133 133 133 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 84 84 77 138 138 127 F F F <--- 10 200-219 83 83 83 137 137 137 F F F 11 220-239 81 81 81 133 133 133 F F F 12 240-259 84 84 81 138 138 133 F F T <--- 13 260-279 83 83 83 137 137 137 F F F 14 280-299 82 82 82 134 134 134 F F F 15 300-319 82 82 82 135 135 135 F F F 16 320-339 84 84 82 138 138 135 F F F <--- 17 340-359 82 82 82 136 136 136 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 81 81 81 133 133 133 T T F <--- 28 560-579 83 83 83 137 137 137 F F F 29 580-599 83 83 83 137 137 137 T T F <--- Max gen: 84, Max flush: 138 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 81 81 81 133 133 133 F F F 1 020-039 85 85 85 139 139 139 F F F 2 040-059 81 81 81 133 133 133 F F F 3 060-079 85 85 85 139 139 139 F F F 4 080-099 85 85 85 139 139 139 F F F 5 100-119 81 81 81 133 133 133 F F F 6 120-139 81 81 81 133 133 133 F F F 7 140-159 81 81 81 133 133 133 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 81 81 81 133 133 133 F F F 12 240-259 84 84 84 138 138 138 F F F 13 260-279 83 83 83 137 137 137 F F F 14 280-299 82 82 82 134 134 134 F F F 15 300-319 82 82 82 135 135 135 F F F 16 320-339 84 84 84 138 138 138 F F F 17 340-359 82 82 82 136 136 136 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 77 77 77 127 127 127 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 F F F 22 440-459 78 78 78 128 128 128 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 85 85 85 139 139 139 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 85, Max flush: 139 Begin loop 43 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.316885296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325021387Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325299093Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.32532484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.32533341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325344809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Crucible 08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc has session id: 74aa6b72-6302-4892-94fe-c4ea36c6cedd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.32567242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Upstairs opts: Upstairs UUID: 08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325703077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Crucible stats registered with UUID: 08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325715756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:13.325725995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325845425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325865173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.325878542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326032039Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326042928Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326069066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326079465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326090144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326100783Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326111232Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326122051Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.32613296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"ds_connection connected from Ok(127.0.0.1:62975)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326198164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"ds_connection connected from Ok(127.0.0.1:45076)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326223122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"ds_connection connected from Ok(127.0.0.1:49922)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326234061Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"io task","client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326807471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326841208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.326911972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327116784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327226465Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[0]R flush_numbers[0..12]: [133, 139, 133, 139, 139, 133, 133, 133, 126, 138, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327250843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[0]R generation[0..12]: [81, 85, 81, 85, 85, 81, 81, 81, 76, 84, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327262282Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327273391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[1]R flush_numbers[0..12]: [133, 139, 133, 139, 139, 133, 133, 133, 126, 138, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.3272839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[1]R generation[0..12]: [81, 85, 81, 85, 85, 81, 81, 81, 76, 84, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327294589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327305588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[2]R flush_numbers[0..12]: [133, 139, 133, 139, 139, 133, 133, 133, 126, 138, 137, 133]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327316227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[2]R generation[0..12]: [81, 85, 81, 85, 85, 81, 81, 81, 76, 84, 83, 81]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327326316Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327337035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327346264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327355703Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327364793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327376102Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"":"downstairs","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327385711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.32739429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"0","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327404009Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"1","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327413488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"client":"2","":"downstairs","session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327422908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327431837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc is now active with session: 74aa6b72-6302-4892-94fe-c4ea36c6cedd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327441526Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327450065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"session_id":"74aa6b72-6302-4892-94fe-c4ea36c6cedd"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327459064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Activated sub_volume 08b5223f-b8e2-4ca7-8a1e-d8b45e5271bc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327466774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327474533Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.327785756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} 01/30 Flush 02/30 Write block 48 len 32768 data: 11 11 9 8 8 7 8 6 03/30 Read block 125 len 32768 04/30 Write block 293 len 4096 data: 8 05/30 Write block 383 len 4096 data: 11 06/30 Write block 7 len 32768 data: 7 8 9 6 6 7 6 5 07/30 Write block 296 len 20480 data: 12 10 10 13 10 08/30 Read block 190 len 24576 09/30 Read block 353 len 24576 10/30 Read block 39 len 20480 11/30 Write block 103 len 36864 data: 8 9 8 7 8 7 6 8 4 12/30 Read block 230 len 24576 13/30 Write block 123 len 40960 data: 9 9 8 9 10 10 9 10 11 9 14/30 Write block 309 len 4096 data: 9 15/30 Read block 196 len 24576 16/30 Read block 125 len 36864 17/30 Flush 18/30 Write block 336 len 40960 data: 7 7 9 9 10 9 7 8 8 9 19/30 Write block 438 len 32768 data: 7 7 8 9 11 10 14 11 20/30 Read block 514 len 40960 21/30 Write block 268 len 24576 data: 9 9 9 8 9 9 22/30 Read block 593 len 28672 23/30 Read block 44 len 20480 24/30 Read block 36 len 20480 25/30 Write block 149 len 40960 data: 7 7 8 7 11 12 11 9 10 8 26/30 Write block 501 len 20480 data: 4 5 4 5 5 27/30 Read block 287 len 24576 28/30 Write block 238 len 8192 data: 10 9 29/30 Write block 230 len 36864 data: 9 8 8 7 8 11 10 11 11 30/30 Read block 584 len 36864 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 23 1 24 Done 29 6 28 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1000)) Some(JobId(1016)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:13.33726871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:13.337477452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1483,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 81 86 141 133 141 F T F <--- 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 81 86 141 133 141 F T F <--- 3 060-079 85 85 85 139 139 139 F F F 4 080-099 85 85 85 139 139 139 F F F 5 100-119 86 81 86 141 133 141 F F F <--- 6 120-139 86 81 86 141 133 141 F F F <--- 7 140-159 81 81 81 133 133 133 T F T <--- 8 160-179 76 76 76 126 126 126 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 81 81 81 133 133 133 T F T <--- 12 240-259 84 84 84 138 138 138 F F F 13 260-279 83 83 83 137 137 137 T F T <--- 14 280-299 86 82 86 141 134 141 F T F <--- 15 300-319 86 82 86 141 135 141 F T F <--- 16 320-339 84 84 84 138 138 138 T F T <--- 17 340-359 82 82 82 136 136 136 T F T <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 86 77 86 141 127 141 F T F <--- 20 400-419 81 81 81 133 133 133 F F F 21 420-439 78 78 78 130 130 130 T F T <--- 22 440-459 78 78 78 128 128 128 T F T <--- 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 73 73 73 120 120 120 T F T <--- 26 520-539 79 79 79 131 131 131 F F F 27 540-559 85 85 85 139 139 139 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 86, Max flush: 141 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 141 141 141 F F F 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 86 86 141 141 141 F F F 3 060-079 85 85 85 139 139 139 F F F 4 080-099 85 85 85 139 139 139 F F F 5 100-119 86 86 86 141 141 141 F F F 6 120-139 86 86 86 141 141 141 F F F 7 140-159 87 87 87 142 142 142 F F F 8 160-179 76 76 76 126 126 126 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 84 84 84 138 138 138 F F F 13 260-279 87 87 87 142 142 142 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 87 87 87 142 142 142 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 86 86 86 141 141 141 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 87 87 87 142 142 142 F F F 22 440-459 87 87 87 142 142 142 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 85 85 85 139 139 139 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 87, Max flush: 142 Begin loop 44 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.598029519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60617215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606475894Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606502961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60651358Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606524989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Crucible 8341c1c0-8857-44df-9331-0dc01991f91a has session id: 2e858450-de32-44e7-a027-e1e90f50e39b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606894177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Upstairs opts: Upstairs UUID: 8341c1c0-8857-44df-9331-0dc01991f91a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606922805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Crucible stats registered with UUID: 8341c1c0-8857-44df-9331-0dc01991f91a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.606933504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:17.606942543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607076921Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60709285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607103019Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607307501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60732405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"8341c1c0-8857-44df-9331-0dc01991f91a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607351527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607360726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607370456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607381715Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607391134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607400883Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607410632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"ds_connection connected from Ok(127.0.0.1:49733)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607523792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"ds_connection connected from Ok(127.0.0.1:40306)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607634603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"ds_connection connected from Ok(127.0.0.1:64722)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.607672899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"io task","client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608157697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608175345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608207983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608488238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608499147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[0]R flush_numbers[0..12]: [141, 139, 141, 139, 139, 141, 141, 142, 126, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608524655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[0]R generation[0..12]: [86, 85, 86, 85, 85, 86, 86, 87, 76, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608535564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608546563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[1]R flush_numbers[0..12]: [141, 139, 141, 139, 139, 141, 141, 142, 126, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608559242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[1]R generation[0..12]: [86, 85, 86, 85, 85, 86, 86, 87, 76, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608569371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60858024Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[2]R flush_numbers[0..12]: [141, 139, 141, 139, 139, 141, 141, 142, 126, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608590849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[2]R generation[0..12]: [86, 85, 86, 85, 85, 86, 86, 87, 76, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608601058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608611807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608621087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608630646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Next flush: 143","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608639845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608649104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"":"downstairs","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608658703Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608667273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"0","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608676672Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"1","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608686131Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"client":"2","":"downstairs","session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.60869681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608705819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"8341c1c0-8857-44df-9331-0dc01991f91a is now active with session: 2e858450-de32-44e7-a027-e1e90f50e39b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608715668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608724258Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"session_id":"2e858450-de32-44e7-a027-e1e90f50e39b"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608733207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Activated sub_volume 8341c1c0-8857-44df-9331-0dc01991f91a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608741326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.608749415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.609080007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} 01/30 Write block 386 len 28672 data: 9 8 9 8 7 10 14 02/30 Write block 277 len 4096 data: 8 03/30 Read block 546 len 40960 04/30 Read block 224 len 20480 05/30 Read block 49 len 32768 06/30 Read block 310 len 20480 07/30 Read block 477 len 32768 08/30 Read block 440 len 20480 09/30 Write block 162 len 20480 data: 6 4 4 4 2 10/30 Write block 128 len 36864 data: 11 10 11 12 10 7 8 8 9 11/30 Write block 345 len 4096 data: 10 12/30 Read block 92 len 24576 13/30 Write block 540 len 36864 data: 5 4 4 4 3 4 3 3 4 14/30 Flush 15/30 Write block 251 len 28672 data: 6 3 5 6 6 6 6 16/30 Read block 153 len 8192 17/30 Read block 553 len 8192 18/30 Read block 316 len 24576 19/30 Write block 253 len 24576 data: 6 7 7 7 7 10 20/30 Read block 311 len 20480 21/30 Read block 282 len 28672 22/30 Read block 69 len 32768 23/30 Read block 483 len 20480 24/30 Flush 25/30 Write block 97 len 24576 data: 4 5 6 6 7 7 26/30 Write block 350 len 36864 data: 11 8 8 6 9 7 7 7 7 27/30 Read block 561 len 36864 28/30 Read block 68 len 4096 29/30 Read block 298 len 12288 30/30 Read block 361 len 32768 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 0 26 Done 30 4 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:17.618027927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:17.618202702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1491,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 141 141 141 F F F 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 86 86 141 141 141 F F F 3 060-079 85 85 85 139 139 139 F F F 4 080-099 85 85 85 139 139 139 T F T <--- 5 100-119 86 86 86 141 141 141 T F T <--- 6 120-139 88 86 88 143 141 143 F F F <--- 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 76 88 143 126 143 F F F <--- 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 88 84 88 144 138 144 F F F <--- 13 260-279 88 87 88 143 142 143 F T F <--- 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 88 87 88 143 142 143 T F T <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 88 86 88 143 141 143 F T F <--- 20 400-419 81 81 81 133 133 133 F F F 21 420-439 87 87 87 142 142 142 F F F 22 440-459 87 87 87 142 142 142 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 88 85 88 143 139 143 F F F <--- 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 88, Max flush: 144 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 141 141 141 F F F 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 86 86 141 141 141 F F F 3 060-079 85 85 85 139 139 139 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 89 89 89 145 145 145 F F F 6 120-139 88 88 88 143 143 143 F F F 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 88 88 143 143 143 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 83 83 137 137 137 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 88 88 88 144 144 144 F F F 13 260-279 88 88 88 143 143 143 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 89 89 89 145 145 145 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 88 88 88 143 143 143 F F F 20 400-419 81 81 81 133 133 133 F F F 21 420-439 87 87 87 142 142 142 F F F 22 440-459 87 87 87 142 142 142 F F F 23 460-479 83 83 83 137 137 137 F F F 24 480-499 81 81 81 133 133 133 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 88 88 88 143 143 143 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 89, Max flush: 145 Begin loop 45 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.867411838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:21.874627039Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.874683444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.881687354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.881941972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.881968219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.881976479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.881987758Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Crucible a39ab870-ceac-4519-854a-32e959475dd1 has session id: 70afacef-6bc8-4edf-8263-b613f7dd4ee6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882289281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Upstairs opts: Upstairs UUID: a39ab870-ceac-4519-854a-32e959475dd1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.88231138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Crucible stats registered with UUID: a39ab870-ceac-4519-854a-32e959475dd1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882321209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:21.882329948Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882471436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882502063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882512612Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882640241Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.88265589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"a39ab870-ceac-4519-854a-32e959475dd1 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882679257Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882688377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882697896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882707365Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882716604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882726393Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882736233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:52488)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882781589Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:63497)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882792348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:50920)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.882802337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"io task","client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883400475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883417193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883484847Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883773392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883811769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[0]R flush_numbers[0..12]: [141, 139, 141, 139, 145, 145, 143, 142, 143, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883848316Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[0]R generation[0..12]: [86, 85, 86, 85, 89, 89, 88, 87, 88, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883859105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883869854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[1]R flush_numbers[0..12]: [141, 139, 141, 139, 145, 145, 143, 142, 143, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883880243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[1]R generation[0..12]: [86, 85, 86, 85, 89, 89, 88, 87, 88, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883890402Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883900981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[2]R flush_numbers[0..12]: [141, 139, 141, 139, 145, 145, 143, 142, 143, 138, 137, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.88391385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[2]R generation[0..12]: [86, 85, 86, 85, 89, 89, 88, 87, 88, 84, 83, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883924159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883934858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883943917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883953356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Next flush: 146","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883962466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883971655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"":"downstairs","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883981124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883989673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"0","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.883999063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"1","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884008452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"client":"2","":"downstairs","session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884018211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.88402706Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"a39ab870-ceac-4519-854a-32e959475dd1 is now active with session: 70afacef-6bc8-4edf-8263-b613f7dd4ee6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884036899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884045748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884053078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"session_id":"70afacef-6bc8-4edf-8263-b613f7dd4ee6"} {"msg":"Activated sub_volume a39ab870-ceac-4519-854a-32e959475dd1","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884063087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884070976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.884344102Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} 01/30 Write block 453 len 32768 data: 7 7 7 7 5 5 4 6 02/30 Flush 03/30 Write block 547 len 40960 data: 4 5 4 4 4 6 5 5 8 7 04/30 Flush 05/30 Write block 201 len 16384 data: 2 3 4 4 06/30 Read block 56 len 32768 07/30 Write block 498 len 8192 data: 6 6 08/30 Read block 133 len 32768 09/30 Write block 9 len 28672 data: 10 7 7 8 7 6 8 10/30 Read block 112 len 36864 11/30 Write block 414 len 36864 data: 7 7 11 9 11 8 9 8 7 12/30 Write block 395 len 32768 data: 10 7 7 11 9 10 11 12 13/30 Read block 486 len 20480 14/30 Write block 111 len 40960 data: 5 5 6 5 6 7 6 7 7 7 15/30 Read block 304 len 40960 16/30 Read block 300 len 20480 17/30 Read block 458 len 28672 18/30 Write block 546 len 24576 data: 4 5 6 5 5 5 19/30 Read block 12 len 36864 20/30 Read block 583 len 40960 21/30 Flush 22/30 Write block 63 len 36864 data: 6 5 5 5 7 8 8 8 8 23/30 Read block 288 len 36864 24/30 Read block 175 len 32768 25/30 Read block 262 len 20480 26/30 Read block 223 len 20480 27/30 Write block 202 len 32768 data: 4 5 5 6 6 6 6 7 28/30 Write block 351 len 8192 data: 9 9 29/30 Read block 84 len 40960 30/30 Read block 364 len 12288 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Read 5 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 3 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:21.894653154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:21.89481877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1499,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 90 90 141 148 148 F F F <--- 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 86 86 141 141 141 F F F 3 060-079 85 85 85 139 139 139 F T T <--- 4 080-099 89 89 89 145 145 145 F F F 5 100-119 89 90 90 145 148 148 F F F <--- 6 120-139 88 90 90 143 148 148 F F F <--- 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 88 88 143 143 143 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 83 90 90 137 148 148 F T T <--- 11 220-239 87 87 87 142 142 142 F F F 12 240-259 88 88 88 144 144 144 F F F 13 260-279 88 88 88 143 143 143 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 89 89 89 145 145 145 F T T <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 88 90 90 143 148 148 F F F <--- 20 400-419 81 90 90 133 148 148 F F F <--- 21 420-439 87 90 90 142 148 148 F F F <--- 22 440-459 87 90 90 142 146 146 F F F <--- 23 460-479 83 90 90 137 146 146 F F F <--- 24 480-499 81 90 90 133 148 148 F F F <--- 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 88 90 90 143 148 148 F F F <--- 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 90, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 148 148 148 F F F 1 020-039 85 85 85 139 139 139 F F F 2 040-059 86 86 86 141 141 141 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 89 89 89 145 145 145 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 90 90 90 148 148 148 F F F 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 88 88 143 143 143 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 91 91 91 149 149 149 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 88 88 88 144 144 144 F F F 13 260-279 88 88 88 143 143 143 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 90 90 90 148 148 148 F F F 20 400-419 90 90 90 148 148 148 F F F 21 420-439 90 90 90 148 148 148 F F F 22 440-459 90 90 90 146 146 146 F F F 23 460-479 90 90 90 146 146 146 F F F 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 90 90 90 148 148 148 F F F 28 560-579 83 83 83 137 137 137 F F F 29 580-599 85 85 85 139 139 139 F F F Max gen: 91, Max flush: 149 Begin loop 46 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.155627386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.163906174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164204978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164228626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164236865Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164248274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Crucible 5418dcb2-e44b-4aaf-b9c5-906301a16242 has session id: ab4bebea-13dc-4f4f-af56-0892c0a3a88c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164597844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Upstairs opts: Upstairs UUID: 5418dcb2-e44b-4aaf-b9c5-906301a16242, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164616042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Crucible stats registered with UUID: 5418dcb2-e44b-4aaf-b9c5-906301a16242","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164626052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:26.164635011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164780528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164812845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164823734Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164949233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164964742Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"5418dcb2-e44b-4aaf-b9c5-906301a16242 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.16498777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.164997189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165006878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165016418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165025807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165035486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165045145Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"ds_connection connected from Ok(127.0.0.1:33684)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.16510584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"ds_connection connected from Ok(127.0.0.1:59826)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165139917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"ds_connection connected from Ok(127.0.0.1:32812)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165226139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"io task","client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165721406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.165734525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.16579528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166067276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166076615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[0]R flush_numbers[0..12]: [148, 139, 141, 149, 145, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166093804Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[0]R generation[0..12]: [90, 85, 86, 91, 89, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166104173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166114912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[1]R flush_numbers[0..12]: [148, 139, 141, 149, 145, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166125231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[1]R generation[0..12]: [90, 85, 86, 91, 89, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.16613545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166146139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[2]R flush_numbers[0..12]: [148, 139, 141, 149, 145, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166156528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[2]R generation[0..12]: [90, 85, 86, 91, 89, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166166597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166177216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166186396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166195775Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166206704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166215913Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"":"downstairs","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166225772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166234481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"0","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166244211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"1","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.16625379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"client":"2","":"downstairs","session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166263659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166272648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"5418dcb2-e44b-4aaf-b9c5-906301a16242 is now active with session: ab4bebea-13dc-4f4f-af56-0892c0a3a88c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166282447Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166290996Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"session_id":"ab4bebea-13dc-4f4f-af56-0892c0a3a88c"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166299806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Activated sub_volume 5418dcb2-e44b-4aaf-b9c5-906301a16242","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166307605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166315534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.166644366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} 01/30 Write block 246 len 24576 data: 9 11 11 11 8 7 02/30 Read block 445 len 8192 03/30 Write block 44 len 4096 data: 7 04/30 Read block 205 len 28672 05/30 Write block 90 len 20480 data: 8 5 6 6 5 06/30 Write block 387 len 24576 data: 9 10 9 8 11 15 07/30 Write block 474 len 8192 data: 11 12 08/30 Read block 75 len 40960 09/30 Write block 569 len 24576 data: 7 4 6 6 5 5 10/30 Write block 392 len 12288 data: 16 10 11 11/30 Read block 282 len 4096 12/30 Read block 417 len 16384 13/30 Read block 443 len 40960 14/30 Flush 15/30 Write block 440 len 12288 data: 9 10 12 16/30 Read block 520 len 4096 17/30 Write block 462 len 12288 data: 6 8 7 18/30 Write block 553 len 4096 data: 6 19/30 Read block 144 len 24576 20/30 Read block 93 len 16384 21/30 Write block 581 len 8192 data: 3 5 22/30 Write block 431 len 40960 data: 9 9 8 7 6 6 7 8 8 10 23/30 Read block 89 len 8192 24/30 Write block 470 len 4096 data: 7 25/30 Write block 22 len 16384 data: 4 4 6 6 26/30 Write block 389 len 8192 data: 10 9 27/30 Read block 110 len 8192 28/30 Read block 262 len 24576 29/30 Read block 561 len 36864 30/30 Read block 573 len 24576 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Write 1 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 0 30 Done 29 1 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) None Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:26.174595243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:26.174775777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1507,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 148 148 148 F F F 1 020-039 85 85 85 139 139 139 T F T <--- 2 040-059 92 86 92 150 141 150 F F F <--- 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 89 92 150 145 150 F F F <--- 5 100-119 90 90 90 148 148 148 F F F 6 120-139 90 90 90 148 148 148 F F F 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 88 88 143 143 143 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 91 91 91 149 149 149 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 92 88 92 150 144 150 F T F <--- 13 260-279 88 88 88 143 143 143 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 92 90 92 150 148 150 T F T <--- 20 400-419 90 90 90 148 148 148 F F F 21 420-439 90 90 90 148 148 148 T F T <--- 22 440-459 90 90 90 146 146 146 T F T <--- 23 460-479 92 90 92 150 146 150 T F T <--- 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 90 90 90 148 148 148 T F T <--- 28 560-579 92 83 92 150 137 150 F F F <--- 29 580-599 85 85 85 139 139 139 T F T <--- Max gen: 92, Max flush: 150 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 148 148 148 F F F 1 020-039 93 93 93 151 151 151 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 90 90 90 148 148 148 F F F 7 140-159 87 87 87 142 142 142 F F F 8 160-179 88 88 88 143 143 143 F F F 9 180-199 84 84 84 138 138 138 F F F 10 200-219 91 91 91 149 149 149 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 92 92 92 150 150 150 F F F 13 260-279 88 88 88 143 143 143 F F F 14 280-299 86 86 86 141 141 141 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 93 93 93 151 151 151 F F F 20 400-419 90 90 90 148 148 148 F F F 21 420-439 93 93 93 151 151 151 F F F 22 440-459 93 93 93 151 151 151 F F F 23 460-479 93 93 93 151 151 151 F F F 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 93 93 93 151 151 151 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 93, Max flush: 151 Begin loop 47 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.424521567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.432740451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433020036Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433045924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433054303Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433065762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Crucible 83dfa452-32e0-4b46-95f5-b3ac3899c3ca has session id: 98e0c8d1-5033-414a-bd2d-0b3d880502e0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433410552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Upstairs opts: Upstairs UUID: 83dfa452-32e0-4b46-95f5-b3ac3899c3ca, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.43343364Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Crucible stats registered with UUID: 83dfa452-32e0-4b46-95f5-b3ac3899c3ca","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433443749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:30.433452849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433597156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433647452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.43366138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.4337754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433793689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"83dfa452-32e0-4b46-95f5-b3ac3899c3ca active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433817577Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433826956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433838125Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433847534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433856833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433866503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433876202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:48954)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433921858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:43404)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433932697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:38725)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.433961564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"io task","client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.434615507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.434672072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.434716508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.43493167Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.434970806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[0]R flush_numbers[0..12]: [148, 151, 150, 149, 150, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.434990455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[0]R generation[0..12]: [90, 93, 92, 91, 92, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435001104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435013623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[1]R flush_numbers[0..12]: [148, 151, 150, 149, 150, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435024112Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[1]R generation[0..12]: [90, 93, 92, 91, 92, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435034111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.43504486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[2]R flush_numbers[0..12]: [148, 151, 150, 149, 150, 148, 148, 142, 143, 138, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435055309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[2]R generation[0..12]: [90, 93, 92, 91, 92, 90, 90, 87, 88, 84, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435065268Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435076007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435085316Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435094826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Next flush: 152","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435103925Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435113074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"":"downstairs","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435122603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435131072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"0","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435140462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"1","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435149951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"client":"2","":"downstairs","session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.43516047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435169359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"83dfa452-32e0-4b46-95f5-b3ac3899c3ca is now active with session: 98e0c8d1-5033-414a-bd2d-0b3d880502e0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435179078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435187647Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"session_id":"98e0c8d1-5033-414a-bd2d-0b3d880502e0"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435196447Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Activated sub_volume 83dfa452-32e0-4b46-95f5-b3ac3899c3ca","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435204396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435212485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.435572914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} 01/30 Write block 156 len 8192 data: 10 11 02/30 Write block 8 len 32768 data: 9 11 8 8 9 8 7 9 03/30 Read block 391 len 40960 04/30 Write block 449 len 40960 data: 10 12 10 6 8 8 8 8 6 6 05/30 Write block 258 len 8192 data: 11 10 06/30 Write block 416 len 36864 data: 12 10 12 9 10 9 8 7 6 07/30 Read block 410 len 28672 08/30 Read block 575 len 28672 09/30 Read block 347 len 8192 10/30 Flush 11/30 Write block 13 len 40960 data: 9 8 10 7 7 6 7 5 4 5 12/30 Read block 274 len 12288 13/30 Write block 179 len 8192 data: 3 4 14/30 Read block 395 len 24576 15/30 Read block 294 len 12288 16/30 Flush 17/30 Write block 132 len 8192 data: 11 8 18/30 Read block 176 len 12288 19/30 Read block 304 len 32768 20/30 Read block 41 len 24576 21/30 Read block 177 len 36864 22/30 Read block 364 len 16384 23/30 Write block 544 len 12288 data: 4 5 5 24/30 Read block 110 len 28672 25/30 Write block 280 len 12288 data: 9 8 7 26/30 Read block 499 len 12288 27/30 Read block 61 len 36864 28/30 Read block 41 len 4096 29/30 Write block 260 len 12288 data: 10 12 10 30/30 Write block 334 len 20480 data: 7 7 8 8 10 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 10 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 27 31 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:30.444065314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:30.444247878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1515,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 90 153 153 148 F F T <--- 1 020-039 94 94 93 153 153 151 F F F <--- 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 90 90 90 148 148 148 T T F <--- 7 140-159 94 94 87 152 152 142 F F T <--- 8 160-179 94 94 88 153 153 143 F F F <--- 9 180-199 94 94 84 153 153 138 F F F <--- 10 200-219 91 91 91 149 149 149 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 94 94 92 152 152 150 F F F <--- 13 260-279 88 88 88 143 143 143 T T F <--- 14 280-299 86 86 86 141 141 141 T T F <--- 15 300-319 86 86 86 141 141 141 F F F 16 320-339 87 87 87 142 142 142 T T F <--- 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 93 93 93 151 151 151 F F F 20 400-419 94 94 90 152 152 148 F F F <--- 21 420-439 94 94 93 152 152 151 F F F <--- 22 440-459 94 94 93 152 152 151 F F F <--- 23 460-479 93 93 93 151 151 151 F F F 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 93 93 93 151 151 151 T T F <--- 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 94, Max flush: 153 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 94 94 94 153 153 153 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 95 95 95 154 154 154 F F F 7 140-159 94 94 94 152 152 152 F F F 8 160-179 94 94 94 153 153 153 F F F 9 180-199 94 94 94 153 153 153 F F F 10 200-219 91 91 91 149 149 149 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 94 94 94 152 152 152 F F F 13 260-279 95 95 95 154 154 154 F F F 14 280-299 95 95 95 154 154 154 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 95 95 95 154 154 154 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 93 93 93 151 151 151 F F F 20 400-419 94 94 94 152 152 152 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 94 94 94 152 152 152 F F F 23 460-479 93 93 93 151 151 151 F F F 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 F F F 26 520-539 79 79 79 131 131 131 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 95, Max flush: 154 Begin loop 48 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.699903873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708157943Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708454348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708487955Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708496924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708508463Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Crucible 6e6d82fb-145c-469e-a235-40b431176fc5 has session id: 0c0564dc-cc16-4fc8-91b0-902baee979ad","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708861632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Upstairs opts: Upstairs UUID: 6e6d82fb-145c-469e-a235-40b431176fc5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.70889229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Crucible stats registered with UUID: 6e6d82fb-145c-469e-a235-40b431176fc5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.708903279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:34.708912458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709053655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709086433Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709097772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.7092256Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709237079Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"6e6d82fb-145c-469e-a235-40b431176fc5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709262037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709271876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709281796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709291735Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709301564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709311933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709322132Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:59422)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709446661Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:44922)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709523944Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:39190)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.709541063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"io task","client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710099264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710116923Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.71014443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710427726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710437515Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[0]R flush_numbers[0..12]: [153, 153, 150, 149, 150, 148, 154, 152, 153, 153, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710455153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[0]R generation[0..12]: [94, 94, 92, 91, 92, 90, 95, 94, 94, 94, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710465372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710476212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[1]R flush_numbers[0..12]: [153, 153, 150, 149, 150, 148, 154, 152, 153, 153, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710486661Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[1]R generation[0..12]: [94, 94, 92, 91, 92, 90, 95, 94, 94, 94, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.71049686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710507769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[2]R flush_numbers[0..12]: [153, 153, 150, 149, 150, 148, 154, 152, 153, 153, 149, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710518068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[2]R generation[0..12]: [94, 94, 92, 91, 92, 90, 95, 94, 94, 94, 91, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710530077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710540916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710550225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710559814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Next flush: 155","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710569003Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710578273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"":"downstairs","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710587782Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710596411Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"0","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.7106059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"1","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710615389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"client":"2","":"downstairs","session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710625229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710634108Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"6e6d82fb-145c-469e-a235-40b431176fc5 is now active with session: 0c0564dc-cc16-4fc8-91b0-902baee979ad","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710643817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710652356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"session_id":"0c0564dc-cc16-4fc8-91b0-902baee979ad"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710661275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Activated sub_volume 6e6d82fb-145c-469e-a235-40b431176fc5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710669135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.710678094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.711030853Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} 01/30 Write block 531 len 24576 data: 5 6 7 9 8 8 02/30 Write block 408 len 12288 data: 6 6 7 03/30 Write block 285 len 4096 data: 7 04/30 Write block 184 len 16384 data: 7 7 6 6 05/30 Write block 458 len 40960 data: 7 5 7 6 7 9 8 8 7 6 06/30 Read block 465 len 12288 07/30 Flush 08/30 Write block 252 len 32768 data: 4 7 8 8 8 8 12 11 09/30 Read block 515 len 40960 10/30 Write block 128 len 8192 data: 12 11 11/30 Read block 48 len 4096 12/30 Write block 504 len 40960 data: 6 6 8 9 7 8 8 8 11 12 13/30 Read block 210 len 36864 14/30 Write block 458 len 32768 data: 8 6 8 7 8 10 9 9 15/30 Write block 155 len 36864 data: 12 11 12 9 8 7 6 7 5 16/30 Read block 129 len 32768 17/30 Read block 184 len 24576 18/30 Read block 322 len 20480 19/30 Read block 74 len 24576 20/30 Read block 528 len 20480 21/30 Write block 528 len 20480 data: 7 6 5 6 7 22/30 Read block 346 len 12288 23/30 Read block 530 len 32768 24/30 Read block 4 len 4096 25/30 Read block 115 len 12288 26/30 Read block 65 len 36864 27/30 Write block 401 len 32768 data: 12 13 11 12 9 8 8 7 28/30 Write block 211 len 12288 data: 10 9 8 29/30 Write block 33 len 12288 data: 8 9 10 30/30 Read block 95 len 36864 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Write 4 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Read 3 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 25 26 Done 30 29 5 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1006)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:34.71967829Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:34.719860704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1523,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 94 94 94 153 153 153 T T F <--- 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 95 95 95 154 154 154 T T F <--- 7 140-159 94 94 94 152 152 152 T T F <--- 8 160-179 94 94 94 153 153 153 T T F <--- 9 180-199 96 96 94 155 155 153 F F T <--- 10 200-219 91 91 91 149 149 149 T T F <--- 11 220-239 87 87 87 142 142 142 F F F 12 240-259 94 94 94 152 152 152 T T F <--- 13 260-279 95 95 95 154 154 154 F F F 14 280-299 96 96 95 155 155 154 F F T <--- 15 300-319 86 86 86 141 141 141 F F F 16 320-339 95 95 95 154 154 154 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 93 93 93 151 151 151 F F F 20 400-419 96 96 94 155 155 152 T T T <--- 21 420-439 94 94 94 152 152 152 F F F 22 440-459 96 96 94 155 155 152 T T T <--- 23 460-479 96 96 93 155 155 151 T T T <--- 24 480-499 90 90 90 148 148 148 F F F 25 500-519 87 87 87 142 142 142 T T F <--- 26 520-539 96 96 79 155 155 131 T T T <--- 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 96, Max flush: 155 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 97 97 97 156 156 156 F F F 7 140-159 97 97 97 156 156 156 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 97 97 97 156 156 156 F F F 11 220-239 87 87 87 142 142 142 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 95 95 95 154 154 154 F F F 14 280-299 96 96 96 155 155 155 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 95 95 95 154 154 154 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 93 93 93 151 151 151 F F F 20 400-419 97 97 97 156 156 156 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 97 97 97 156 156 156 F F F 23 460-479 97 97 97 156 156 156 F F F 24 480-499 90 90 90 148 148 148 F F F 25 500-519 97 97 97 156 156 156 F F F 26 520-539 97 97 97 156 156 156 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 97, Max flush: 156 Begin loop 49 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.980444479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.988568521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.988847887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.988874255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.988882744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.988893993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Crucible 5c943544-01f3-4820-b07b-8848be85e8c0 has session id: c363bd64-c6d5-44da-9d20-d2fc0cb8040a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989247262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Upstairs opts: Upstairs UUID: 5c943544-01f3-4820-b07b-8848be85e8c0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989264441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Crucible stats registered with UUID: 5c943544-01f3-4820-b07b-8848be85e8c0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.98927422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:38.989295058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989444845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989475822Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989488461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.98961795Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989636418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"5c943544-01f3-4820-b07b-8848be85e8c0 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989653287Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989662106Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989671585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989681054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989690444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989700223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989709952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"ds_connection connected from Ok(127.0.0.1:37957)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989772596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50879)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989838251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45041)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.989897616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"io task","client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990442438Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990477125Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990547899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990753821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990787878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[0]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990835074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[0]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990846443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990857352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[1]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990867831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[1]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990889059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[2]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 142]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990899548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[2]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 87]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990909687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990920366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990929766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990939295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Next flush: 157","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990948544Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990959993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"":"downstairs","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990969682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990978201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"0","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.990987691Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"1","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991008419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"client":"2","":"downstairs","session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991018158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991027277Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"5c943544-01f3-4820-b07b-8848be85e8c0 is now active with session: c363bd64-c6d5-44da-9d20-d2fc0cb8040a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991036876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991045376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"session_id":"c363bd64-c6d5-44da-9d20-d2fc0cb8040a"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991054185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Activated sub_volume 5c943544-01f3-4820-b07b-8848be85e8c0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.991062064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-09-18T03:31:38.991069953Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:38.99134182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} 01/30 Flush 02/30 Write block 524 len 32768 data: 9 9 9 9 8 7 6 7 03/30 Read block 227 len 24576 04/30 Flush 05/30 Write block 327 len 20480 data: 11 11 9 8 7 06/30 Write block 387 len 20480 data: 10 11 11 10 12 07/30 Read block 7 len 28672 08/30 Read block 408 len 28672 09/30 Read block 558 len 32768 10/30 Write block 459 len 4096 data: 7 11/30 Read block 119 len 32768 12/30 Read block 283 len 12288 13/30 Read block 278 len 32768 14/30 Read block 222 len 28672 15/30 Flush 16/30 Write block 225 len 8192 data: 5 7 17/30 Write block 522 len 12288 data: 8 6 10 18/30 Write block 444 len 16384 data: 15 12 13 11 19/30 Read block 287 len 20480 20/30 Read block 164 len 36864 21/30 Read block 545 len 36864 22/30 Read block 245 len 36864 23/30 Read block 577 len 32768 24/30 Write block 392 len 40960 data: 17 11 12 11 8 8 12 10 11 13 25/30 Read block 279 len 16384 26/30 Read block 126 len 24576 27/30 Read block 275 len 8192 28/30 Write block 499 len 36864 data: 7 4 5 6 5 7 7 9 10 29/30 Read block 109 len 24576 30/30 Read block 535 len 12288 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 9 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1000)) Some(JobId(1014)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:39.000455946Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:39.000615952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1531,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 97 97 97 156 156 156 F F F 7 140-159 97 97 97 156 156 156 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 97 97 97 156 156 156 F F F 11 220-239 87 87 87 142 142 142 T F T <--- 12 240-259 97 97 97 156 156 156 F F F 13 260-279 95 95 95 154 154 154 F F F 14 280-299 96 96 96 155 155 155 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 98 95 98 159 154 159 F F F <--- 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 98 93 98 159 151 159 T F T <--- 20 400-419 97 97 97 156 156 156 T F T <--- 21 420-439 94 94 94 152 152 152 F F F 22 440-459 98 97 98 159 156 159 T F T <--- 23 460-479 97 97 97 156 156 156 F F F 24 480-499 90 90 90 148 148 148 T F T <--- 25 500-519 97 97 97 156 156 156 T F T <--- 26 520-539 98 97 98 158 156 158 T T T <--- 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 98, Max flush: 159 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 97 97 97 156 156 156 F F F 7 140-159 97 97 97 156 156 156 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 97 97 97 156 156 156 F F F 11 220-239 99 99 99 160 160 160 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 95 95 95 154 154 154 F F F 14 280-299 96 96 96 155 155 155 F F F 15 300-319 86 86 86 141 141 141 F F F 16 320-339 98 98 98 159 159 159 F F F 17 340-359 91 91 91 149 149 149 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F F F 20 400-419 99 99 99 160 160 160 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 97 97 97 156 156 156 F F F 24 480-499 99 99 99 160 160 160 F F F 25 500-519 99 99 99 160 160 160 F F F 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 93 93 151 151 151 F F F Max gen: 99, Max flush: 160 Begin loop 50 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.24670053Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:43.253892494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.253942009Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.260985336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261248923Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.26127837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261286849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261298088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Crucible e5807db8-15db-4934-95d7-f465a5f8f93a has session id: 39d53cd9-b631-473e-a0ad-e154a6ab6e11","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261678315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Upstairs opts: Upstairs UUID: e5807db8-15db-4934-95d7-f465a5f8f93a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261708873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Crucible stats registered with UUID: e5807db8-15db-4934-95d7-f465a5f8f93a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261719772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:43.261728981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261840531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.2618576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.261868059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262030095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262053663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"e5807db8-15db-4934-95d7-f465a5f8f93a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262069861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.2620786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262089489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262098849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262108178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262117837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262127676Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"ds_connection connected from Ok(127.0.0.1:60866)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262189171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"ds_connection connected from Ok(127.0.0.1:43098)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262215318Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"ds_connection connected from Ok(127.0.0.1:57225)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262226567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"io task","client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262824565Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262862902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.262874191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263122209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263140998Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[0]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 160]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.26323324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[0]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 99]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263244359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263257348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[1]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 160]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263267787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[1]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 99]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263277896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263288525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[2]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 156, 156, 156, 155, 156, 160]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263299084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[2]R generation[0..12]: [94, 97, 92, 91, 92, 90, 97, 97, 97, 96, 97, 99]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263309183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263319842Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263329121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263338711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Next flush: 161","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.26334784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263356889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"":"downstairs","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263366648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263375217Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"0","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263384767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"1","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263394216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"client":"2","":"downstairs","session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263405225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263414184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"e5807db8-15db-4934-95d7-f465a5f8f93a is now active with session: 39d53cd9-b631-473e-a0ad-e154a6ab6e11","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263424023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263432622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"session_id":"39d53cd9-b631-473e-a0ad-e154a6ab6e11"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263441372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Activated sub_volume e5807db8-15db-4934-95d7-f465a5f8f93a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.263449251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.2634573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.26380607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} 01/30 Write block 278 len 4096 data: 10 02/30 Write block 585 len 4096 data: 10 03/30 Read block 453 len 32768 04/30 Write block 330 len 40960 data: 9 8 6 7 8 8 9 9 11 10 05/30 Read block 337 len 16384 06/30 Write block 128 len 12288 data: 13 12 12 07/30 Read block 574 len 16384 08/30 Write block 316 len 28672 data: 8 8 8 11 11 10 12 09/30 Read block 508 len 12288 10/30 Read block 318 len 24576 11/30 Write block 510 len 20480 data: 9 9 12 13 10 12/30 Read block 402 len 24576 13/30 Write block 491 len 4096 data: 7 14/30 Read block 221 len 20480 15/30 Read block 308 len 8192 16/30 Read block 133 len 28672 17/30 Read block 232 len 4096 18/30 Write block 210 len 20480 data: 8 11 10 9 7 19/30 Read block 588 len 16384 20/30 Read block 335 len 8192 21/30 Flush 22/30 Write block 231 len 20480 data: 9 9 8 9 12 23/30 Read block 231 len 16384 24/30 Write block 478 len 12288 data: 11 8 6 25/30 Read block 484 len 24576 26/30 Read block 255 len 16384 27/30 Write block 143 len 36864 data: 6 8 8 7 9 6 8 8 9 28/30 Read block 21 len 32768 29/30 Read block 51 len 40960 30/30 Write block 349 len 28672 data: 9 12 10 10 7 10 8 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:43.2727584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:43.272909797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1539,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 97 100 100 156 161 161 F F F <--- 7 140-159 97 97 97 156 156 156 F T T <--- 8 160-179 97 97 97 156 156 156 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 97 100 100 156 161 161 F F F <--- 11 220-239 99 99 99 160 160 160 F T T <--- 12 240-259 97 97 97 156 156 156 F F F 13 260-279 95 100 100 154 161 161 T F F <--- 14 280-299 96 96 96 155 155 155 F F F 15 300-319 86 100 100 141 161 161 F F F <--- 16 320-339 98 100 100 159 161 161 T F F <--- 17 340-359 91 91 91 149 149 149 F T T <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F F F 20 400-419 99 99 99 160 160 160 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 97 97 97 156 156 156 F T T <--- 24 480-499 99 100 100 160 161 161 F T T <--- 25 500-519 99 100 100 160 161 161 F F F <--- 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 93 100 100 151 161 161 T F F <--- Max gen: 100, Max flush: 161 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 100 100 100 161 161 161 F F F 7 140-159 101 101 101 162 162 162 F F F 8 160-179 97 97 97 156 156 156 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 100 100 100 161 161 161 F F F 14 280-299 96 96 96 155 155 155 F F F 15 300-319 100 100 100 161 161 161 F F F 16 320-339 100 100 100 161 161 161 F F F 17 340-359 101 101 101 162 162 162 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F F F 20 400-419 99 99 99 160 160 160 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 101 101 101 162 162 162 F F F 25 500-519 100 100 100 161 161 161 F F F 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 101, Max flush: 162 Begin loop 51 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.53030547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:47.537504912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.537559948Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.544426419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.544687137Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.544712205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.544720764Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.544732373Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Crucible 89025d80-53f4-45c6-aecf-e6f28a5b65bd has session id: eb5fb481-a004-4c51-ad56-c8bdfba16bea","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545098821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Upstairs opts: Upstairs UUID: 89025d80-53f4-45c6-aecf-e6f28a5b65bd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545127218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Crucible stats registered with UUID: 89025d80-53f4-45c6-aecf-e6f28a5b65bd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545137557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:47.545146507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545272586Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545288984Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545298973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545444321Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545460719Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"89025d80-53f4-45c6-aecf-e6f28a5b65bd active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545483997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545493246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545502886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545512245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545521784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545531673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545543342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"ds_connection connected from Ok(127.0.0.1:57100)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545580819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"ds_connection connected from Ok(127.0.0.1:61174)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545591628Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"ds_connection connected from Ok(127.0.0.1:41954)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.545601647Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"io task","client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546293557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546327824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546338953Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.54659919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546640507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[0]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 161, 162, 156, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546687292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[0]R generation[0..12]: [94, 97, 92, 91, 92, 90, 100, 101, 97, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546698281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546709091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[1]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 161, 162, 156, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.54671949Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[1]R generation[0..12]: [94, 97, 92, 91, 92, 90, 100, 101, 97, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546729679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546742308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[2]R flush_numbers[0..12]: [153, 156, 150, 149, 150, 148, 161, 162, 156, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546752697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[2]R generation[0..12]: [94, 97, 92, 91, 92, 90, 100, 101, 97, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546762886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546773555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546782794Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546792333Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Next flush: 163","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546801403Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546810602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"":"downstairs","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546820291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.54682885Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"0","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546838379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"1","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546847708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"client":"2","":"downstairs","session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546857288Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546866287Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"89025d80-53f4-45c6-aecf-e6f28a5b65bd is now active with session: eb5fb481-a004-4c51-ad56-c8bdfba16bea","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546876196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546886115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"session_id":"eb5fb481-a004-4c51-ad56-c8bdfba16bea"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546895084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Activated sub_volume 89025d80-53f4-45c6-aecf-e6f28a5b65bd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546902984Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.546911023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.547230395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} 01/30 Write block 330 len 12288 data: 10 9 7 02/30 Write block 165 len 32768 data: 5 3 3 5 5 6 6 6 03/30 Write block 291 len 12288 data: 6 6 9 04/30 Read block 456 len 40960 05/30 Write block 316 len 20480 data: 9 9 9 12 12 06/30 Read block 274 len 8192 07/30 Read block 336 len 24576 08/30 Read block 369 len 32768 09/30 Write block 271 len 24576 data: 9 10 10 9 8 7 10/30 Read block 248 len 8192 11/30 Write block 400 len 40960 data: 12 14 14 12 13 10 9 9 8 7 12/30 Read block 391 len 28672 13/30 Write block 158 len 12288 data: 10 9 8 14/30 Flush 15/30 Flush 16/30 Write block 171 len 8192 data: 7 7 17/30 Read block 115 len 36864 18/30 Flush 19/30 Write block 494 len 12288 data: 5 5 3 20/30 Read block 248 len 40960 21/30 Read block 361 len 24576 22/30 Read block 245 len 8192 23/30 Read block 538 len 8192 24/30 Write block 569 len 4096 data: 8 25/30 Read block 449 len 32768 26/30 Write block 280 len 8192 data: 10 9 27/30 Read block 369 len 8192 28/30 Read block 34 len 40960 29/30 Write block 6 len 8192 data: 7 8 30/30 Write block 133 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 10 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 2 33 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:47.554832263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:47.555001158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1547,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 153 153 153 F T T <--- 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 100 100 100 161 161 161 F T T <--- 7 140-159 101 102 102 162 163 163 F F F <--- 8 160-179 97 102 102 156 165 165 F F F <--- 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 100 102 102 161 163 163 F F F <--- 14 280-299 96 102 102 155 163 163 F T T <--- 15 300-319 100 102 102 161 163 163 F F F <--- 16 320-339 100 102 102 161 163 163 T F F <--- 17 340-359 101 101 101 162 162 162 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F F F 20 400-419 99 102 102 160 163 163 F F F <--- 21 420-439 94 94 94 152 152 152 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 101 101 101 162 162 162 F T T <--- 25 500-519 100 100 100 161 161 161 F F F 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 92 92 92 150 150 150 F T T <--- 29 580-599 100 100 100 161 161 161 F F F Max gen: 102, Max flush: 165 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 103 103 103 166 166 166 F F F 1 020-039 97 97 97 156 156 156 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 103 103 103 166 166 166 F F F 7 140-159 102 102 102 163 163 163 F F F 8 160-179 102 102 102 165 165 165 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 102 102 102 163 163 163 F F F 14 280-299 103 103 103 166 166 166 F F F 15 300-319 102 102 102 163 163 163 F F F 16 320-339 102 102 102 163 163 163 F F F 17 340-359 101 101 101 162 162 162 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F F F 20 400-419 102 102 102 163 163 163 F F F 21 420-439 94 94 94 152 152 152 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 103 103 103 166 166 166 F F F 25 500-519 100 100 100 161 161 161 F F F 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 103 103 103 166 166 166 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 103, Max flush: 166 Begin loop 52 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.807495548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:51.814692681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.814747566Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.821653094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.821909592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.82193561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.821944269Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.821955708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Crucible 53e75845-49fe-48c9-aacf-8bc4ab0b215d has session id: f78f8640-dd14-45bc-b333-116cd57a1259","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.82227232Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Upstairs opts: Upstairs UUID: 53e75845-49fe-48c9-aacf-8bc4ab0b215d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822301768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Crucible stats registered with UUID: 53e75845-49fe-48c9-aacf-8bc4ab0b215d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822315037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:51.822324096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822442276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822472293Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822483002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822610411Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822629149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"53e75845-49fe-48c9-aacf-8bc4ab0b215d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822656917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822666176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822675655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822685094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822694534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822704133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822714002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"ds_connection connected from Ok(127.0.0.1:59353)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822761898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"ds_connection connected from Ok(127.0.0.1:39978)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822772647Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"ds_connection connected from Ok(127.0.0.1:40512)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.822812463Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"io task","client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823414831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823435449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823552069Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823742212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.82376697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[0]R flush_numbers[0..12]: [166, 156, 150, 149, 150, 148, 166, 163, 165, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823819976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[0]R generation[0..12]: [103, 97, 92, 91, 92, 90, 103, 102, 102, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823831005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823841944Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[1]R flush_numbers[0..12]: [166, 156, 150, 149, 150, 148, 166, 163, 165, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823852413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[1]R generation[0..12]: [103, 97, 92, 91, 92, 90, 103, 102, 102, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823862712Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823873361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[2]R flush_numbers[0..12]: [166, 156, 150, 149, 150, 148, 166, 163, 165, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.82388374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[2]R generation[0..12]: [103, 97, 92, 91, 92, 90, 103, 102, 102, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823893989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823904878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823915957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823925566Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Next flush: 167","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823934756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823943985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"":"downstairs","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823953504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823962133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"0","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823971522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"1","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823980931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"client":"2","":"downstairs","session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.823990721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.82399985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"53e75845-49fe-48c9-aacf-8bc4ab0b215d is now active with session: f78f8640-dd14-45bc-b333-116cd57a1259","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824009749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824018348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"session_id":"f78f8640-dd14-45bc-b333-116cd57a1259"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824027377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Activated sub_volume 53e75845-49fe-48c9-aacf-8bc4ab0b215d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824035227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824043336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.824397085Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} 01/30 Flush 02/30 Write block 26 len 32768 data: 6 6 6 4 4 7 7 9 03/30 Read block 66 len 36864 04/30 Read block 396 len 36864 05/30 Read block 64 len 24576 06/30 Write block 266 len 24576 data: 12 10 10 10 10 10 07/30 Write block 496 len 16384 data: 4 5 7 8 08/30 Read block 417 len 12288 09/30 Write block 0 len 16384 data: 2 3 5 6 10/30 Write block 128 len 8192 data: 14 13 11/30 Write block 323 len 28672 data: 12 12 11 12 12 12 10 12/30 Read block 176 len 20480 13/30 Write block 174 len 24576 data: 5 4 5 5 3 4 14/30 Read block 102 len 20480 15/30 Write block 412 len 12288 data: 5 5 8 16/30 Read block 580 len 8192 17/30 Read block 112 len 32768 18/30 Write block 387 len 40960 data: 11 12 12 11 13 18 12 13 12 9 19/30 Write block 61 len 4096 data: 7 20/30 Read block 557 len 20480 21/30 Write block 496 len 28672 data: 5 6 8 9 5 6 7 22/30 Read block 302 len 32768 23/30 Write block 506 len 16384 data: 10 11 8 9 24/30 Write block 434 len 20480 data: 8 7 7 8 9 25/30 Read block 327 len 4096 26/30 Read block 347 len 40960 27/30 Write block 341 len 12288 data: 10 8 9 28/30 Read block 353 len 36864 29/30 Read block 333 len 24576 30/30 Write block 389 len 36864 data: 13 12 14 19 13 14 13 10 9 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:51.833509181Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:51.833725133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1555,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 103 103 103 166 166 166 F T T <--- 1 020-039 97 97 97 156 156 156 T T T 2 040-059 92 92 92 150 150 150 F F F 3 060-079 91 91 91 149 149 149 F T T <--- 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 103 103 103 166 166 166 F T T <--- 7 140-159 102 102 102 163 163 163 F F F 8 160-179 102 102 102 165 165 165 F T T <--- 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 102 102 102 163 163 163 F T T <--- 14 280-299 103 103 103 166 166 166 F F F 15 300-319 102 102 102 163 163 163 F F F 16 320-339 102 102 102 163 163 163 F T T <--- 17 340-359 101 101 101 162 162 162 F T T <--- 18 360-379 81 81 81 133 133 133 F F F 19 380-399 99 99 99 160 160 160 F T T <--- 20 400-419 102 102 102 163 163 163 F T T <--- 21 420-439 94 94 94 152 152 152 F T T <--- 22 440-459 99 99 99 160 160 160 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 103 103 103 166 166 166 F T T <--- 25 500-519 100 100 100 161 161 161 F T T <--- 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 103 103 103 166 166 166 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 103, Max flush: 166 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 104 104 104 167 167 167 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 104 104 104 167 167 167 F F F 7 140-159 102 102 102 163 163 163 F F F 8 160-179 104 104 104 167 167 167 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 104 104 104 167 167 167 F F F 14 280-299 103 103 103 166 166 166 F F F 15 300-319 102 102 102 163 163 163 F F F 16 320-339 104 104 104 167 167 167 F F F 17 340-359 104 104 104 167 167 167 F F F 18 360-379 81 81 81 133 133 133 F F F 19 380-399 104 104 104 167 167 167 F F F 20 400-419 104 104 104 167 167 167 F F F 21 420-439 104 104 104 167 167 167 F F F 22 440-459 99 99 99 160 160 160 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 104 104 104 167 167 167 F F F 25 500-519 104 104 104 167 167 167 F F F 26 520-539 99 99 99 160 160 160 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 103 103 103 166 166 166 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 104, Max flush: 167 Begin loop 53 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.099833037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108011504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108300449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108324297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108332556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108343985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Crucible f2dfd398-8e25-4660-932e-85713c5878de has session id: d1f476ef-0c6d-41d8-9481-7e8a49cc5813","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.10874523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Upstairs opts: Upstairs UUID: f2dfd398-8e25-4660-932e-85713c5878de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108774418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Crucible stats registered with UUID: f2dfd398-8e25-4660-932e-85713c5878de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108785197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:56.108794396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108916845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108934564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.108944373Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109114908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109130547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"f2dfd398-8e25-4660-932e-85713c5878de active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109166184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109175323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109184852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109194211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.10920351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109213439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109223089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"ds_connection connected from Ok(127.0.0.1:50192)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109314701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"ds_connection connected from Ok(127.0.0.1:33465)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109375105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"ds_connection connected from Ok(127.0.0.1:57452)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109386204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"io task","client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.109997161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110022659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110076424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110308504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110344061Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[0]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110394646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[0]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110405786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110416625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[1]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110427224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[1]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110437863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110448582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[2]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110458931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[2]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.11046917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110480099Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Max found gen is 105","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110489368Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Generation requested: 106 >= found:105","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110498927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Next flush: 168","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110508057Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110517136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"":"downstairs","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110526675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110536604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"0","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110546153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"1","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110555602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"client":"2","":"downstairs","session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110565222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110574161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"f2dfd398-8e25-4660-932e-85713c5878de is now active with session: d1f476ef-0c6d-41d8-9481-7e8a49cc5813","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.11058386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110592359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"session_id":"d1f476ef-0c6d-41d8-9481-7e8a49cc5813"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110601338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Activated sub_volume f2dfd398-8e25-4660-932e-85713c5878de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110609138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110617097Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.110947198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} 01/30 Write block 337 len 40960 data: 10 12 11 11 11 9 10 9 11 9 02/30 Read block 448 len 32768 03/30 Read block 265 len 16384 04/30 Write block 261 len 28672 data: 13 11 9 10 12 13 11 05/30 Write block 310 len 32768 data: 10 7 7 7 8 8 10 10 06/30 Read block 52 len 4096 07/30 Write block 404 len 12288 data: 14 11 10 08/30 Write block 529 len 36864 data: 8 7 8 8 8 10 9 9 8 09/30 Write block 399 len 20480 data: 11 13 15 15 13 10/30 Read block 389 len 20480 11/30 Read block 478 len 28672 12/30 Read block 188 len 16384 13/30 Read block 87 len 28672 14/30 Read block 230 len 40960 15/30 Read block 242 len 28672 16/30 Write block 374 len 32768 data: 11 12 13 14 11 10 10 12 17/30 Flush 18/30 Write block 482 len 4096 data: 9 19/30 Read block 297 len 36864 20/30 Read block 35 len 24576 21/30 Read block 105 len 28672 22/30 Write block 565 len 12288 data: 7 6 5 23/30 Write block 333 len 28672 data: 8 9 9 10 11 13 12 24/30 Write block 383 len 20480 data: 12 10 9 10 12 25/30 Read block 486 len 24576 26/30 Read block 159 len 36864 27/30 Read block 582 len 36864 28/30 Write block 451 len 36864 data: 11 7 9 9 9 9 7 9 8 29/30 Read block 17 len 16384 30/30 Read block 92 len 40960 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 8 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:31:56.12033954Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:31:56.120514345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1563,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 104 104 104 167 167 167 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 104 104 104 167 167 167 F F F 7 140-159 102 102 102 163 163 163 F F F 8 160-179 104 104 104 167 167 167 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 106 106 104 168 168 167 F F F <--- 14 280-299 103 103 103 166 166 166 F F F 15 300-319 106 106 102 168 168 163 F F F <--- 16 320-339 106 106 104 168 168 167 T T T <--- 17 340-359 106 106 104 168 168 167 F F T <--- 18 360-379 106 106 81 168 168 133 F F F <--- 19 380-399 106 106 104 168 168 167 T T F <--- 20 400-419 106 106 104 168 168 167 F F F <--- 21 420-439 104 104 104 167 167 167 F F F 22 440-459 99 99 99 160 160 160 T T F <--- 23 460-479 101 101 101 162 162 162 F F F 24 480-499 104 104 104 167 167 167 T T F <--- 25 500-519 104 104 104 167 167 167 F F F 26 520-539 106 106 99 168 168 160 F F F <--- 27 540-559 95 95 95 154 154 154 F F F 28 560-579 103 103 103 166 166 166 T T F <--- 29 580-599 100 100 100 161 161 161 F F F Max gen: 106, Max flush: 168 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 104 104 104 167 167 167 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 90 90 148 148 148 F F F 6 120-139 104 104 104 167 167 167 F F F 7 140-159 102 102 102 163 163 163 F F F 8 160-179 104 104 104 167 167 167 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F F F 13 260-279 106 106 106 168 168 168 F F F 14 280-299 103 103 103 166 166 166 F F F 15 300-319 106 106 106 168 168 168 F F F 16 320-339 107 107 107 169 169 169 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 106 106 106 168 168 168 F F F 19 380-399 107 107 107 169 169 169 F F F 20 400-419 106 106 106 168 168 168 F F F 21 420-439 104 104 104 167 167 167 F F F 22 440-459 107 107 107 169 169 169 F F F 23 460-479 101 101 101 162 162 162 F F F 24 480-499 107 107 107 169 169 169 F F F 25 500-519 104 104 104 167 167 167 F F F 26 520-539 106 106 106 168 168 168 F F F 27 540-559 95 95 95 154 154 154 F F F 28 560-579 107 107 107 169 169 169 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 107, Max flush: 169 Begin loop 54 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.374587857Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:00.381774341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.381829316Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.388833816Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389087554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389112542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389123321Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.38913461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Crucible 957d90c8-63ff-440f-9ddb-fc4f535b17cf has session id: 7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389467591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Upstairs opts: Upstairs UUID: 957d90c8-63ff-440f-9ddb-fc4f535b17cf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389489149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Crucible stats registered with UUID: 957d90c8-63ff-440f-9ddb-fc4f535b17cf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389498998Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:00.389507847Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389657714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389673623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389683532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.38982659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389843028Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"957d90c8-63ff-440f-9ddb-fc4f535b17cf active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389877425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389886594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389895994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389907063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389916332Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.389926121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.38993576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:56842)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.390005254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:38607)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.390030292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"ds_connection connected from Ok(127.0.0.1:55903)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.390041431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"io task","client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.390711192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.39073798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.390748909Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391020926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391056122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[0]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391103778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[0]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391114717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391125596Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[1]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391138165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[1]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391148394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391159193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[2]R flush_numbers[0..12]: [167, 167, 150, 167, 150, 148, 167, 163, 167, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391169493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[2]R generation[0..12]: [104, 104, 92, 104, 92, 90, 104, 102, 104, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391179832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391190541Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.39119975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391209479Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Next flush: 170","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391218738Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391227997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"":"downstairs","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391237737Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391246466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"0","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391256035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"1","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391265474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"client":"2","":"downstairs","session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391275163Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391285192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"957d90c8-63ff-440f-9ddb-fc4f535b17cf is now active with session: 7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391294942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391303491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"session_id":"7edb0df9-4bcb-4a0f-a1bb-661719c4b7e6"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.39131243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Activated sub_volume 957d90c8-63ff-440f-9ddb-fc4f535b17cf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391320519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391328449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.391668119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} 01/30 Write block 22 len 16384 data: 6 5 7 7 02/30 Write block 118 len 4096 data: 8 03/30 Read block 249 len 12288 04/30 Read block 579 len 16384 05/30 Write block 304 len 36864 data: 13 11 10 10 10 10 11 8 8 06/30 Write block 417 len 16384 data: 11 13 10 11 07/30 Read block 279 len 28672 08/30 Write block 534 len 40960 data: 11 10 10 9 7 6 6 5 5 5 09/30 Write block 103 len 12288 data: 9 10 9 10/30 Read block 534 len 20480 11/30 Read block 31 len 8192 12/30 Write block 516 len 4096 data: 9 13/30 Read block 82 len 8192 14/30 Flush 15/30 Write block 365 len 32768 data: 11 10 11 13 12 8 9 9 16/30 Read block 110 len 36864 17/30 Read block 104 len 36864 18/30 Write block 448 len 20480 data: 11 11 13 12 8 19/30 Write block 362 len 16384 data: 11 12 11 12 20/30 Flush 21/30 Write block 475 len 40960 data: 13 10 10 12 9 7 9 10 10 9 22/30 Flush 23/30 Write block 290 len 16384 data: 5 7 7 10 24/30 Write block 127 len 12288 data: 11 15 14 25/30 Read block 550 len 36864 26/30 Read block 313 len 32768 27/30 Read block 322 len 20480 28/30 Write block 254 len 4096 data: 9 29/30 Write block 162 len 16384 data: 8 6 5 6 30/30 Read block 286 len 28672 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Write 1 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 7 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:00.400563624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:00.400738809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1571,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 104 108 108 167 170 170 F F F <--- 2 040-059 92 92 92 150 150 150 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 90 108 108 148 170 170 F F F <--- 6 120-139 104 104 104 167 167 167 F T T <--- 7 140-159 102 102 102 163 163 163 F F F 8 160-179 104 104 104 167 167 167 F T T <--- 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 97 97 97 156 156 156 F T T <--- 13 260-279 106 106 106 168 168 168 F F F 14 280-299 103 103 103 166 166 166 F T T <--- 15 300-319 106 108 108 168 170 170 F F F <--- 16 320-339 107 107 107 169 169 169 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 106 108 108 168 171 171 F F F <--- 19 380-399 107 107 107 169 169 169 F F F 20 400-419 106 108 108 168 170 170 F F F <--- 21 420-439 104 108 108 167 170 170 F F F <--- 22 440-459 107 108 108 169 171 171 F F F <--- 23 460-479 101 108 108 162 172 172 F F F <--- 24 480-499 107 108 108 169 172 172 F F F <--- 25 500-519 104 108 108 167 170 170 F F F <--- 26 520-539 106 108 108 168 170 170 F F F <--- 27 540-559 95 108 108 154 170 170 F F F <--- 28 560-579 107 107 107 169 169 169 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 108, Max flush: 172 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 92 92 92 150 150 150 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 92 92 92 150 150 150 F F F 5 100-119 108 108 108 170 170 170 F F F 6 120-139 109 109 109 173 173 173 F F F 7 140-159 102 102 102 163 163 163 F F F 8 160-179 109 109 109 173 173 173 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 100 100 100 161 161 161 F F F 11 220-239 101 101 101 162 162 162 F F F 12 240-259 109 109 109 173 173 173 F F F 13 260-279 106 106 106 168 168 168 F F F 14 280-299 109 109 109 173 173 173 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 107 107 107 169 169 169 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 107 107 107 169 169 169 F F F 20 400-419 108 108 108 170 170 170 F F F 21 420-439 108 108 108 170 170 170 F F F 22 440-459 108 108 108 171 171 171 F F F 23 460-479 108 108 108 172 172 172 F F F 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 F F F 26 520-539 108 108 108 170 170 170 F F F 27 540-559 108 108 108 170 170 170 F F F 28 560-579 107 107 107 169 169 169 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 109, Max flush: 173 Begin loop 55 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.664413265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.67273573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673018355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673048573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673057242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673068601Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Crucible e53d4dc7-1def-4cf2-9a34-5ae527ec1f62 has session id: f97a0686-3434-4717-8dce-1bc798172dbd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673380024Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Upstairs opts: Upstairs UUID: e53d4dc7-1def-4cf2-9a34-5ae527ec1f62, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673409611Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Crucible stats registered with UUID: e53d4dc7-1def-4cf2-9a34-5ae527ec1f62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.6734225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:04.673431989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673566337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673618013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673640531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673722334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673745242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"e53d4dc7-1def-4cf2-9a34-5ae527ec1f62 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673781749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673791318Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673801297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673810866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673820295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673830074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673840064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:51679)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673896599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:47696)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673914957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:60891)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.673925586Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"io task","client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.67456454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674594588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674606427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674866854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674900501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[0]R flush_numbers[0..12]: [167, 170, 150, 167, 150, 170, 173, 163, 173, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674947567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[0]R generation[0..12]: [104, 108, 92, 104, 92, 108, 109, 102, 109, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674958536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674969445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[1]R flush_numbers[0..12]: [167, 170, 150, 167, 150, 170, 173, 163, 173, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674979914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[1]R generation[0..12]: [104, 108, 92, 104, 92, 108, 109, 102, 109, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.674990103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675000702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[2]R flush_numbers[0..12]: [167, 170, 150, 167, 150, 170, 173, 163, 173, 155, 161, 162]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675011122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[2]R generation[0..12]: [104, 108, 92, 104, 92, 108, 109, 102, 109, 96, 100, 101]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.67502354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.67503435Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675043639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675053188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Next flush: 174","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675062477Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675071656Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"":"downstairs","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675086575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675095134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"0","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675104553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"1","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675113913Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"client":"2","":"downstairs","session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675123452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675132381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"e53d4dc7-1def-4cf2-9a34-5ae527ec1f62 is now active with session: f97a0686-3434-4717-8dce-1bc798172dbd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.67514221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675150649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"session_id":"f97a0686-3434-4717-8dce-1bc798172dbd"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675159689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Activated sub_volume e53d4dc7-1def-4cf2-9a34-5ae527ec1f62","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675167498Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675175397Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.675497469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} 01/30 Write block 406 len 36864 data: 11 10 9 8 8 6 6 6 9 02/30 Read block 76 len 12288 03/30 Read block 575 len 4096 04/30 Write block 88 len 32768 data: 10 10 9 6 7 7 6 4 05/30 Read block 403 len 28672 06/30 Read block 339 len 8192 07/30 Read block 308 len 32768 08/30 Write block 337 len 12288 data: 12 14 13 09/30 Read block 114 len 40960 10/30 Flush 11/30 Write block 279 len 24576 data: 11 11 10 8 7 7 12/30 Read block 359 len 4096 13/30 Flush 14/30 Write block 148 len 24576 data: 7 9 9 10 8 12 15/30 Write block 201 len 4096 data: 3 16/30 Read block 46 len 8192 17/30 Read block 549 len 12288 18/30 Flush 19/30 Write block 537 len 28672 data: 10 8 7 7 6 6 6 20/30 Write block 44 len 20480 data: 8 7 8 8 12 21/30 Read block 423 len 32768 22/30 Write block 226 len 24576 data: 8 9 10 9 10 10 23/30 Read block 468 len 28672 24/30 Read block 349 len 12288 25/30 Read block 145 len 4096 26/30 Flush 27/30 Write block 81 len 36864 data: 8 6 5 5 6 8 7 11 11 28/30 Write block 438 len 32768 data: 10 9 11 11 13 11 16 13 29/30 Read block 135 len 36864 30/30 Read block 325 len 12288 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 3 Done Done Sent false 1002 Acked Read 1 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:04.684593147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:04.684757042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1579,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 110 110 92 177 177 150 F F F <--- 3 060-079 104 104 104 167 167 167 F F F 4 080-099 110 110 92 174 174 150 T T F <--- 5 100-119 108 108 108 170 170 170 F F F 6 120-139 109 109 109 173 173 173 F F F 7 140-159 110 110 102 176 176 163 F F F <--- 8 160-179 109 109 109 173 173 173 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 110 110 100 176 176 161 F F F <--- 11 220-239 110 110 101 177 177 162 F F F <--- 12 240-259 109 109 109 173 173 173 F F F 13 260-279 110 110 106 175 175 168 F F F <--- 14 280-299 110 110 109 175 175 173 F F F <--- 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 107 174 174 169 F F F <--- 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 107 107 107 169 169 169 F F F 20 400-419 110 110 108 174 174 170 F F T <--- 21 420-439 108 108 108 170 170 170 T T F <--- 22 440-459 108 108 108 171 171 171 T T F <--- 23 460-479 108 108 108 172 172 172 F F F 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 F F F 26 520-539 110 110 108 177 177 170 F F F <--- 27 540-559 110 110 108 177 177 170 F F F <--- 28 560-579 107 107 107 169 169 169 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 110, Max flush: 177 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 110 110 110 177 177 177 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 108 108 108 170 170 170 F F F 6 120-139 109 109 109 173 173 173 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 109 109 109 173 173 173 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 110 110 110 176 176 176 F F F 11 220-239 110 110 110 177 177 177 F F F 12 240-259 109 109 109 173 173 173 F F F 13 260-279 110 110 110 175 175 175 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 107 107 107 169 169 169 F F F 20 400-419 110 110 110 174 174 174 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 111 111 178 178 178 F F F 23 460-479 108 108 108 172 172 172 F F F 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 F F F 26 520-539 110 110 110 177 177 177 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 107 107 107 169 169 169 F F F 29 580-599 100 100 100 161 161 161 F F F Max gen: 111, Max flush: 178 Begin loop 56 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.939932469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:08.947080386Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.947122412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954141511Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954386909Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954416067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954424486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954435675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Crucible 2087061b-2db9-4b94-9bb9-3895f106880a has session id: fd77cb72-1fbb-46eb-9074-5e6d50eeae03","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954779585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Upstairs opts: Upstairs UUID: 2087061b-2db9-4b94-9bb9-3895f106880a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954801143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Crucible stats registered with UUID: 2087061b-2db9-4b94-9bb9-3895f106880a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954810652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:08.954819522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954941581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954959889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.954969819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955126995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955150973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"2087061b-2db9-4b94-9bb9-3895f106880a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955164982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955173581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95518328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955192879Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955202168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955231366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955254104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"ds_connection connected from Ok(127.0.0.1:39453)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955265063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"ds_connection connected from Ok(127.0.0.1:57053)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95530254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"ds_connection connected from Ok(127.0.0.1:54098)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955358975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955830634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.955915366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956030846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956195372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95621309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[0]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 170, 173, 176, 173, 155, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956306252Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[0]R generation[0..12]: [104, 108, 110, 104, 111, 108, 109, 110, 109, 96, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956317481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95632831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[1]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 170, 173, 176, 173, 155, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956338659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[1]R generation[0..12]: [104, 108, 110, 104, 111, 108, 109, 110, 109, 96, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956348908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956359457Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[2]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 170, 173, 176, 173, 155, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956369807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[2]R generation[0..12]: [104, 108, 110, 104, 111, 108, 109, 110, 109, 96, 110, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956380196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956390965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956400164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956409743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Next flush: 179","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956420842Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956430121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"":"downstairs","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95643953Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.95644803Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"0","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956457509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"1","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956467228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"client":"2","":"downstairs","session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956477197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956486326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"2087061b-2db9-4b94-9bb9-3895f106880a is now active with session: fd77cb72-1fbb-46eb-9074-5e6d50eeae03","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956496046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956504545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"session_id":"fd77cb72-1fbb-46eb-9074-5e6d50eeae03"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956513474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Activated sub_volume 2087061b-2db9-4b94-9bb9-3895f106880a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956521333Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956529163Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.956847805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} 01/30 Write block 134 len 24576 data: 9 9 10 9 8 10 02/30 Read block 527 len 16384 03/30 Read block 292 len 36864 04/30 Write block 211 len 12288 data: 12 11 10 05/30 Write block 252 len 4096 data: 5 06/30 Write block 120 len 8192 data: 8 7 07/30 Write block 270 len 24576 data: 11 11 11 11 10 9 08/30 Write block 111 len 16384 data: 6 6 7 6 09/30 Write block 402 len 32768 data: 16 14 15 12 12 11 10 9 10/30 Write block 261 len 40960 data: 14 12 10 11 13 14 12 11 11 12 11/30 Write block 473 len 12288 data: 6 12 14 12/30 Write block 386 len 28672 data: 11 13 13 14 13 15 20 13/30 Flush 14/30 Write block 383 len 20480 data: 13 11 10 12 14 15/30 Read block 64 len 40960 16/30 Flush 17/30 Write block 163 len 20480 data: 7 6 7 4 4 18/30 Read block 402 len 32768 19/30 Read block 563 len 40960 20/30 Read block 491 len 20480 21/30 Read block 8 len 24576 22/30 Read block 591 len 12288 23/30 Write block 572 len 28672 data: 7 6 6 6 5 3 2 24/30 Write block 410 len 28672 data: 9 7 7 7 10 8 13 25/30 Read block 265 len 8192 26/30 Read block 264 len 28672 27/30 Read block 249 len 12288 28/30 Read block 22 len 16384 29/30 Read block 399 len 20480 30/30 Write block 576 len 20480 data: 6 4 3 2 3 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Sent Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:08.965360373Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:08.965537888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1587,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 110 110 110 177 177 177 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 108 112 112 170 179 179 F F F <--- 6 120-139 109 112 112 173 179 179 F F F <--- 7 140-159 110 110 110 176 176 176 F F F 8 160-179 109 109 109 173 173 173 F T T <--- 9 180-199 96 96 96 155 155 155 F F F 10 200-219 110 112 112 176 179 179 F F F <--- 11 220-239 110 110 110 177 177 177 F F F 12 240-259 109 112 112 173 179 179 F F F <--- 13 260-279 110 112 112 175 179 179 F F F <--- 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 107 112 112 169 180 180 F F F <--- 20 400-419 110 112 112 174 179 179 F T T <--- 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 111 111 178 178 178 F F F 23 460-479 108 112 112 172 179 179 F F F <--- 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 F F F 26 520-539 110 110 110 177 177 177 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 107 107 107 169 169 169 F T T <--- 29 580-599 100 100 100 161 161 161 F T T <--- Max gen: 112, Max flush: 180 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 110 110 110 177 177 177 F F F 3 060-079 104 104 104 167 167 167 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 110 110 110 176 176 176 F F F 8 160-179 113 113 113 181 181 181 F F F 9 180-199 96 96 96 155 155 155 F F F 10 200-219 112 112 112 179 179 179 F F F 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 112 112 112 179 179 179 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 112 112 112 180 180 180 F F F 20 400-419 113 113 113 181 181 181 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 111 111 178 178 178 F F F 23 460-479 112 112 112 179 179 179 F F F 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 F F F 26 520-539 110 110 110 177 177 177 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 113 113 113 181 181 181 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 113, Max flush: 181 Begin loop 57 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.219437315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.227772469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228067653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.22810193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.22811131Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228123159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Crucible beb7afd9-cf87-437b-990e-53b6a3fa5976 has session id: c34e36b5-67d3-4589-9088-4ddee024d576","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228432642Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Upstairs opts: Upstairs UUID: beb7afd9-cf87-437b-990e-53b6a3fa5976, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.22845617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Crucible stats registered with UUID: beb7afd9-cf87-437b-990e-53b6a3fa5976","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228467119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:13.228476238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228609906Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228626185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228636214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228780951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228804859Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"beb7afd9-cf87-437b-990e-53b6a3fa5976 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228829797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228838956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228850395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228859834Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228869344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228878943Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.22890702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"ds_connection connected from Ok(127.0.0.1:39140)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228933158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"ds_connection connected from Ok(127.0.0.1:45468)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228943947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"ds_connection connected from Ok(127.0.0.1:37933)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.228973445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.229568733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.229587451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.229679193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.229902714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.229938071Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[0]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 179, 179, 176, 181, 155, 179, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230008044Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[0]R generation[0..12]: [104, 108, 110, 104, 111, 112, 112, 110, 113, 96, 112, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230019243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230032292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[1]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 179, 179, 176, 181, 155, 179, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230042841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[1]R generation[0..12]: [104, 108, 110, 104, 111, 112, 112, 110, 113, 96, 112, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.23005308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.23006393Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[2]R flush_numbers[0..12]: [167, 170, 177, 167, 178, 179, 179, 176, 181, 155, 179, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230074359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[2]R generation[0..12]: [104, 108, 110, 104, 111, 112, 112, 110, 113, 96, 112, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230084678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230095427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230104646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230114275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Next flush: 182","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230123424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230132684Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"":"downstairs","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230142293Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230150802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"0","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230160161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"1","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.23017088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"client":"2","":"downstairs","session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230180489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230189379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"beb7afd9-cf87-437b-990e-53b6a3fa5976 is now active with session: c34e36b5-67d3-4589-9088-4ddee024d576","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230199088Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230207607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"session_id":"c34e36b5-67d3-4589-9088-4ddee024d576"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230216606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Activated sub_volume beb7afd9-cf87-437b-990e-53b6a3fa5976","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230224336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230232225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.230546128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} 01/30 Write block 566 len 4096 data: 7 02/30 Write block 46 len 24576 data: 9 9 13 12 10 9 03/30 Write block 470 len 16384 data: 8 6 6 7 04/30 Flush 05/30 Write block 58 len 40960 data: 5 7 6 8 5 7 6 6 6 8 06/30 Read block 10 len 24576 07/30 Read block 59 len 24576 08/30 Read block 370 len 24576 09/30 Write block 520 len 16384 data: 8 8 9 7 10/30 Read block 541 len 40960 11/30 Read block 149 len 32768 12/30 Write block 191 len 32768 data: 8 8 5 5 7 6 6 5 13/30 Read block 14 len 32768 14/30 Flush 15/30 Write block 142 len 20480 data: 7 7 9 9 8 16/30 Read block 343 len 4096 17/30 Read block 379 len 36864 18/30 Read block 203 len 36864 19/30 Read block 579 len 4096 20/30 Flush 21/30 Write block 145 len 4096 data: 10 22/30 Read block 555 len 16384 23/30 Write block 404 len 36864 data: 16 13 13 12 11 10 10 8 8 24/30 Read block 428 len 20480 25/30 Write block 205 len 40960 data: 7 7 7 7 8 9 13 12 11 8 26/30 Read block 200 len 24576 27/30 Read block 593 len 20480 28/30 Write block 519 len 20480 data: 9 9 9 10 8 29/30 Read block 455 len 20480 30/30 Write block 516 len 36864 data: 10 9 8 10 10 10 11 9 11 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Write 6 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:13.239687501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:13.239859076Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1595,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 114 114 110 183 183 177 F F F <--- 3 060-079 114 114 104 183 183 167 F F F <--- 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 114 114 110 184 184 176 T T F <--- 8 160-179 113 113 113 181 181 181 F F F 9 180-199 114 114 96 183 183 155 F F F <--- 10 200-219 112 112 112 179 179 179 T T F <--- 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 112 112 112 179 179 179 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 112 112 112 180 180 180 F F F 20 400-419 113 113 113 181 181 181 T T F <--- 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 111 111 178 178 178 F F F 23 460-479 114 114 112 182 182 179 F F F <--- 24 480-499 108 108 108 172 172 172 F F F 25 500-519 108 108 108 170 170 170 T T F <--- 26 520-539 114 114 110 183 183 177 T T F <--- 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 113 182 182 181 F F F <--- 29 580-599 113 113 113 181 181 181 F F F Max gen: 114, Max flush: 184 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 167 167 167 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 114 114 114 183 183 183 F F F 3 060-079 114 114 114 183 183 183 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 115 115 115 185 185 185 F F F 8 160-179 113 113 113 181 181 181 F F F 9 180-199 114 114 114 183 183 183 F F F 10 200-219 115 115 115 185 185 185 F F F 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 112 112 112 179 179 179 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 112 112 112 180 180 180 F F F 20 400-419 115 115 115 185 185 185 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 111 111 178 178 178 F F F 23 460-479 114 114 114 182 182 182 F F F 24 480-499 108 108 108 172 172 172 F F F 25 500-519 115 115 115 185 185 185 F F F 26 520-539 115 115 115 185 185 185 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 114 182 182 182 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 115, Max flush: 185 Begin loop 58 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.492571296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:17.499719793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.499774599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.506625242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.50688012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.506914747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.506924226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.506935985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Crucible 5d27b571-40d7-45a6-b0ac-7685e5904ccf has session id: 708d8b15-13db-4c18-9e50-e308443dfc6d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507229229Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Upstairs opts: Upstairs UUID: 5d27b571-40d7-45a6-b0ac-7685e5904ccf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507272535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Crucible stats registered with UUID: 5d27b571-40d7-45a6-b0ac-7685e5904ccf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507282904Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:17.507291764Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507418083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507436441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.5074464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507606576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507630294Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"5d27b571-40d7-45a6-b0ac-7685e5904ccf active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507653632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507662721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507672211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.50768156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507690849Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507700528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507711217Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45539)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507743624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:59682)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507753653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:53708)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.507763613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508462902Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.50848728Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508498819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508777774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508787523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[0]R flush_numbers[0..12]: [167, 170, 183, 183, 178, 179, 179, 185, 181, 183, 185, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508806342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[0]R generation[0..12]: [104, 108, 114, 114, 111, 112, 112, 115, 113, 114, 115, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508816751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.5088275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[1]R flush_numbers[0..12]: [167, 170, 183, 183, 178, 179, 179, 185, 181, 183, 185, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508838319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[1]R generation[0..12]: [104, 108, 114, 114, 111, 112, 112, 115, 113, 114, 115, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508848678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508861737Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[2]R flush_numbers[0..12]: [167, 170, 183, 183, 178, 179, 179, 185, 181, 183, 185, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508872336Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[2]R generation[0..12]: [104, 108, 114, 114, 111, 112, 112, 115, 113, 114, 115, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508882635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508893494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508902693Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508912183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Next flush: 186","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508921282Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508930371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"":"downstairs","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.50893981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508948429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"0","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508958129Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"1","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508967678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"client":"2","":"downstairs","session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508977627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508986616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"5d27b571-40d7-45a6-b0ac-7685e5904ccf is now active with session: 708d8b15-13db-4c18-9e50-e308443dfc6d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.508996315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.509006154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"session_id":"708d8b15-13db-4c18-9e50-e308443dfc6d"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.509015074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Activated sub_volume 5d27b571-40d7-45a6-b0ac-7685e5904ccf","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.509022933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.509030792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.509394461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} 01/30 Write block 199 len 8192 data: 3 2 02/30 Read block 338 len 28672 03/30 Write block 50 len 32768 data: 11 10 9 8 9 7 5 6 04/30 Read block 294 len 24576 05/30 Read block 199 len 40960 06/30 Read block 73 len 16384 07/30 Write block 53 len 36864 data: 9 10 8 6 7 6 8 7 9 08/30 Flush 09/30 Write block 455 len 16384 data: 10 10 8 10 10/30 Read block 434 len 24576 11/30 Flush 12/30 Write block 503 len 24576 data: 6 8 8 11 12 9 13/30 Read block 311 len 32768 14/30 Read block 441 len 36864 15/30 Write block 140 len 28672 data: 8 8 8 8 10 11 9 16/30 Write block 512 len 40960 data: 13 14 11 10 11 10 9 11 11 11 17/30 Read block 480 len 16384 18/30 Write block 10 len 8192 data: 9 9 19/30 Write block 518 len 8192 data: 10 12 20/30 Read block 280 len 24576 21/30 Write block 495 len 20480 data: 6 6 7 9 10 22/30 Read block 403 len 8192 23/30 Write block 386 len 12288 data: 13 15 14 24/30 Flush 25/30 Write block 354 len 8192 data: 11 9 26/30 Read block 104 len 24576 27/30 Read block 25 len 16384 28/30 Read block 192 len 40960 29/30 Read block 417 len 40960 30/30 Write block 513 len 24576 data: 15 12 11 12 11 11 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Sent Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:17.518511266Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:17.518727027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1603,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 116 116 167 188 188 F F F <--- 1 020-039 108 108 108 170 170 170 F F F 2 040-059 114 116 116 183 186 186 F F F <--- 3 060-079 114 116 116 183 186 186 F F F <--- 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 115 116 116 185 188 188 F F F <--- 8 160-179 113 113 113 181 181 181 F F F 9 180-199 114 116 116 183 186 186 F F F <--- 10 200-219 115 116 116 185 186 186 F F F <--- 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 112 112 112 179 179 179 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 106 106 106 168 168 168 F T T <--- 18 360-379 108 108 108 171 171 171 F F F 19 380-399 112 116 116 180 188 188 F F F <--- 20 400-419 115 115 115 185 185 185 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 111 116 116 178 187 187 F F F <--- 23 460-479 114 114 114 182 182 182 F F F 24 480-499 108 116 116 172 188 188 F F F <--- 25 500-519 115 116 116 185 188 188 F T T <--- 26 520-539 115 116 116 185 188 188 F F F <--- 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 114 182 182 182 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 116, Max flush: 188 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 188 188 188 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 116 116 116 186 186 186 F F F 3 060-079 116 116 116 186 186 186 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 116 116 116 188 188 188 F F F 8 160-179 113 113 113 181 181 181 F F F 9 180-199 116 116 116 186 186 186 F F F 10 200-219 116 116 116 186 186 186 F F F 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 112 112 112 179 179 179 F F F 14 280-299 110 110 110 175 175 175 F F F 15 300-319 108 108 108 170 170 170 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 108 108 108 171 171 171 F F F 19 380-399 116 116 116 188 188 188 F F F 20 400-419 115 115 115 185 185 185 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 116 116 116 187 187 187 F F F 23 460-479 114 114 114 182 182 182 F F F 24 480-499 116 116 116 188 188 188 F F F 25 500-519 117 117 117 189 189 189 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 114 182 182 182 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 117, Max flush: 189 Begin loop 59 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.770811553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779139888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779441561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779466659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779474968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779486357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Crucible 790a3aaa-b95c-450c-93d2-c73c41a97592 has session id: 2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779845006Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Upstairs opts: Upstairs UUID: 790a3aaa-b95c-450c-93d2-c73c41a97592, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779874254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Crucible stats registered with UUID: 790a3aaa-b95c-450c-93d2-c73c41a97592","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.779885363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:21.779894212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780037759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780067317Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780077996Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780214224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780230952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"790a3aaa-b95c-450c-93d2-c73c41a97592 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.78025412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.7802637Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780273719Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780283388Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780292777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780302646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780312265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"ds_connection connected from Ok(127.0.0.1:42798)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780364801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"ds_connection connected from Ok(127.0.0.1:57231)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780389989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"ds_connection connected from Ok(127.0.0.1:58629)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.780405777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781097197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781122315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781204198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781394581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.78140985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[0]R flush_numbers[0..12]: [188, 170, 186, 186, 178, 179, 179, 188, 181, 186, 186, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781469915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[0]R generation[0..12]: [116, 108, 116, 116, 111, 112, 112, 116, 113, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781480924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781491613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[1]R flush_numbers[0..12]: [188, 170, 186, 186, 178, 179, 179, 188, 181, 186, 186, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781501882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[1]R generation[0..12]: [116, 108, 116, 116, 111, 112, 112, 116, 113, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781512251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.78152292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[2]R flush_numbers[0..12]: [188, 170, 186, 186, 178, 179, 179, 188, 181, 186, 186, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781538119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[2]R generation[0..12]: [116, 108, 116, 116, 111, 112, 112, 116, 113, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781548568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781559137Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781570046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781579515Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Next flush: 190","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781588654Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781597833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"":"downstairs","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781607263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781615782Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"0","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781625221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"1","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.78163453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"client":"2","":"downstairs","session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781644089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781653049Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"790a3aaa-b95c-450c-93d2-c73c41a97592 is now active with session: 2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781662898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781671707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"session_id":"2f9d9bfa-12f7-4024-87ec-d8c1e8b34dfe"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781680616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Activated sub_volume 790a3aaa-b95c-450c-93d2-c73c41a97592","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781688475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.781696375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.782056263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} 01/30 Write block 458 len 28672 data: 11 9 9 8 9 11 10 02/30 Write block 149 len 28672 data: 10 10 11 9 13 13 13 03/30 Read block 214 len 28672 04/30 Write block 270 len 20480 data: 13 12 12 12 11 05/30 Write block 54 len 12288 data: 11 9 7 06/30 Read block 503 len 40960 07/30 Write block 297 len 32768 data: 11 11 14 11 10 12 14 14 08/30 Flush 09/30 Write block 478 len 24576 data: 13 10 8 10 11 11 10/30 Write block 217 len 12288 data: 8 8 10 11/30 Write block 158 len 8192 data: 11 10 12/30 Write block 372 len 36864 data: 10 10 12 13 14 15 12 11 11 13/30 Read block 506 len 40960 14/30 Flush 15/30 Write block 413 len 28672 data: 8 11 9 14 12 14 11 16/30 Read block 565 len 36864 17/30 Write block 484 len 20480 data: 10 7 9 10 9 18/30 Write block 503 len 24576 data: 7 9 9 12 13 10 19/30 Read block 311 len 28672 20/30 Read block 220 len 4096 21/30 Flush 22/30 Write block 399 len 16384 data: 12 14 16 17 23/30 Read block 373 len 36864 24/30 Read block 327 len 40960 25/30 Read block 454 len 28672 26/30 Read block 217 len 12288 27/30 Read block 80 len 32768 28/30 Read block 405 len 8192 29/30 Write block 495 len 24576 data: 7 7 8 10 11 6 30/30 Write block 393 len 12288 data: 14 15 14 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 6 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 22 2 26 Done 20 0 20 40 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1007)) Some(JobId(1020)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:21.79093877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:21.791093206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1611,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 188 188 188 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 118 118 118 190 190 190 F F F 3 060-079 116 116 116 186 186 186 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 118 118 118 191 190 191 F F F <--- 8 160-179 113 113 113 181 181 181 F F F 9 180-199 116 116 116 186 186 186 F F F 10 200-219 118 116 118 191 186 191 F F F <--- 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 118 118 118 190 190 190 F F F 14 280-299 118 118 118 190 190 190 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 108 118 191 171 191 F F F <--- 19 380-399 118 116 118 191 188 191 T F T <--- 20 400-419 118 115 118 192 185 192 T F T <--- 21 420-439 111 111 111 178 178 178 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 118 118 118 191 190 191 F F F <--- 24 480-499 118 116 118 192 188 192 T F T <--- 25 500-519 118 117 118 192 189 192 T F T <--- 26 520-539 116 116 116 188 188 188 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 114 182 182 182 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 118, Max flush: 192 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 188 188 188 F F F 1 020-039 108 108 108 170 170 170 F F F 2 040-059 118 118 118 190 190 190 F F F 3 060-079 116 116 116 186 186 186 F F F 4 080-099 111 111 111 178 178 178 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 112 112 112 179 179 179 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 113 113 113 181 181 181 F F F 9 180-199 116 116 116 186 186 186 F F F 10 200-219 118 118 118 191 191 191 F F F 11 220-239 110 110 110 177 177 177 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 118 118 118 190 190 190 F F F 14 280-299 118 118 118 190 190 190 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 119 119 119 193 193 193 F F F 20 400-419 119 119 119 193 193 193 F F F 21 420-439 111 111 111 178 178 178 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 118 118 118 191 191 191 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 119 119 119 193 193 193 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 110 110 110 177 177 177 F F F 28 560-579 114 114 114 182 182 182 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 119, Max flush: 193 Begin loop 60 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.036710195Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.044891152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045185617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045210874Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045219354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045230813Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Crucible 02ba0b29-8355-4091-8d68-1bc544f121de has session id: d7615d71-63e6-48fc-abcb-711f18c6c61d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045540326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Upstairs opts: Upstairs UUID: 02ba0b29-8355-4091-8d68-1bc544f121de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045568873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Crucible stats registered with UUID: 02ba0b29-8355-4091-8d68-1bc544f121de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045579662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:26.045588682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.0457177Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045735579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045745688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045926072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045937771Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"02ba0b29-8355-4091-8d68-1bc544f121de active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.04595481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045963739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045974778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045984357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.045993786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046003535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046013115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"ds_connection connected from Ok(127.0.0.1:34738)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046085608Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45301)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046107986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"ds_connection connected from Ok(127.0.0.1:35202)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046118235Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046661948Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046689456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046792487Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046943184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.046962802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[0]R flush_numbers[0..12]: [188, 170, 190, 186, 178, 179, 179, 191, 181, 186, 191, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047115659Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[0]R generation[0..12]: [116, 108, 118, 116, 111, 112, 112, 118, 113, 116, 118, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047127158Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047139956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[1]R flush_numbers[0..12]: [188, 170, 190, 186, 178, 179, 179, 191, 181, 186, 191, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047150385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[1]R generation[0..12]: [116, 108, 118, 116, 111, 112, 112, 118, 113, 116, 118, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047160725Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047171424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[2]R flush_numbers[0..12]: [188, 170, 190, 186, 178, 179, 179, 191, 181, 186, 191, 177]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047181783Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[2]R generation[0..12]: [116, 108, 118, 116, 111, 112, 112, 118, 113, 116, 118, 110]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047192092Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047202851Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.04721198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047221739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Next flush: 194","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047230958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047240238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"":"downstairs","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047249937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047258376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"0","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047267815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"1","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047277214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"client":"2","":"downstairs","session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047288383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047297203Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"02ba0b29-8355-4091-8d68-1bc544f121de is now active with session: d7615d71-63e6-48fc-abcb-711f18c6c61d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047306982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047315521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"session_id":"d7615d71-63e6-48fc-abcb-711f18c6c61d"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.04732444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Activated sub_volume 02ba0b29-8355-4091-8d68-1bc544f121de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.04733268Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.047340799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.04767063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} 01/30 Write block 44 len 8192 data: 9 8 02/30 Read block 500 len 12288 03/30 Read block 48 len 28672 04/30 Write block 553 len 12288 data: 7 6 9 05/30 Write block 216 len 24576 data: 8 9 9 11 9 7 06/30 Read block 243 len 8192 07/30 Write block 26 len 32768 data: 7 7 7 5 5 8 8 10 08/30 Read block 21 len 28672 09/30 Write block 540 len 28672 data: 8 7 7 7 5 6 6 10/30 Read block 479 len 20480 11/30 Read block 353 len 16384 12/30 Write block 387 len 16384 data: 16 15 15 14 13/30 Write block 14 len 28672 data: 9 11 8 8 7 8 6 14/30 Flush 15/30 Write block 212 len 40960 data: 13 12 9 9 9 10 10 12 10 8 16/30 Write block 128 len 40960 data: 16 15 13 13 12 10 10 10 11 10 17/30 Write block 572 len 20480 data: 8 7 7 7 7 18/30 Read block 450 len 36864 19/30 Flush 20/30 Write block 176 len 20480 data: 6 6 4 5 5 21/30 Read block 560 len 4096 22/30 Write block 268 len 28672 data: 12 12 14 13 13 13 12 23/30 Read block 44 len 8192 24/30 Write block 568 len 16384 data: 5 9 5 7 25/30 Write block 420 len 24576 data: 12 10 9 8 7 6 26/30 Write block 61 len 28672 data: 10 6 8 7 7 7 9 27/30 Write block 555 len 28672 data: 10 8 7 7 8 8 10 28/30 Read block 1 len 28672 29/30 Write block 63 len 24576 data: 9 8 8 8 10 9 30/30 Write block 83 len 16384 data: 6 6 7 9 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 7 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:26.05616515Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:26.056312947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1619,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 116 194 194 188 F F F <--- 1 020-039 120 120 108 194 194 170 F F F <--- 2 040-059 120 120 118 194 194 190 F F T <--- 3 060-079 116 116 116 186 186 186 T T F <--- 4 080-099 111 111 111 178 178 178 T T F <--- 5 100-119 112 112 112 179 179 179 F F F 6 120-139 120 120 112 195 195 179 F F F <--- 7 140-159 118 118 118 191 191 191 F F F 8 160-179 113 113 113 181 181 181 T T F <--- 9 180-199 116 116 116 186 186 186 T T F <--- 10 200-219 120 120 118 195 195 191 F F F <--- 11 220-239 120 120 110 195 195 177 F F F <--- 12 240-259 112 112 112 179 179 179 F F F 13 260-279 118 118 118 190 190 190 T T F <--- 14 280-299 118 118 118 190 190 190 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 120 120 119 194 194 193 F F F <--- 20 400-419 119 119 119 193 193 193 F F F 21 420-439 111 111 111 178 178 178 T T F <--- 22 440-459 118 118 118 190 190 190 F F F 23 460-479 118 118 118 191 191 191 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 119 119 119 193 193 193 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 120 120 110 194 194 177 T T F <--- 28 560-579 120 120 114 195 195 182 T T F <--- 29 580-599 113 113 113 181 181 181 F F F Max gen: 120, Max flush: 195 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 120 120 120 194 194 194 F F F 2 040-059 120 120 120 194 194 194 F F F 3 060-079 121 121 121 196 196 196 F F F 4 080-099 121 121 121 196 196 196 F F F 5 100-119 112 112 112 179 179 179 F F F 6 120-139 120 120 120 195 195 195 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 121 121 121 196 196 196 F F F 9 180-199 121 121 121 196 196 196 F F F 10 200-219 120 120 120 195 195 195 F F F 11 220-239 120 120 120 195 195 195 F F F 12 240-259 112 112 112 179 179 179 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 118 118 118 190 190 190 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 120 120 120 194 194 194 F F F 20 400-419 119 119 119 193 193 193 F F F 21 420-439 121 121 121 196 196 196 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 118 118 118 191 191 191 F F F 24 480-499 119 119 119 193 193 193 F F F 25 500-519 119 119 119 193 193 193 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 121 121 121 196 196 196 F F F 28 560-579 121 121 121 196 196 196 F F F 29 580-599 113 113 113 181 181 181 F F F Max gen: 121, Max flush: 196 Begin loop 61 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.318568926Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:30.325709394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.32575698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.332779268Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333032366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333060394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333068733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333082442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Crucible fb589f7e-f94b-4791-a26c-170868f716ec has session id: 74d473fc-2c4c-4997-a133-dfb307060498","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333387875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Upstairs opts: Upstairs UUID: fb589f7e-f94b-4791-a26c-170868f716ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333412803Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Crucible stats registered with UUID: fb589f7e-f94b-4791-a26c-170868f716ec","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333422372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:30.333431241Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.33356009Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333591237Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333602036Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333736295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333751283Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"fb589f7e-f94b-4791-a26c-170868f716ec active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333768052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333776761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.33378626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.33379556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333804769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333816078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333825767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"ds_connection connected from Ok(127.0.0.1:59420)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333851395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"ds_connection connected from Ok(127.0.0.1:53132)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333862064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"ds_connection connected from Ok(127.0.0.1:42405)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.333872073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334451282Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334537445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334601609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334746557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334808631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[0]R flush_numbers[0..12]: [194, 194, 194, 196, 196, 179, 195, 191, 196, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334885405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[0]R generation[0..12]: [120, 120, 120, 121, 121, 112, 120, 118, 121, 121, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334896384Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334907193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[1]R flush_numbers[0..12]: [194, 194, 194, 196, 196, 179, 195, 191, 196, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334917652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[1]R generation[0..12]: [120, 120, 120, 121, 121, 112, 120, 118, 121, 121, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334930331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.33494118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[2]R flush_numbers[0..12]: [194, 194, 194, 196, 196, 179, 195, 191, 196, 196, 195, 195]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334951649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[2]R generation[0..12]: [120, 120, 120, 121, 121, 112, 120, 118, 121, 121, 120, 120]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334962058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334972607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334981886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.334991415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Next flush: 197","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335000595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335009754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"":"downstairs","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335019283Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335027842Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"0","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335037361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"1","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.3350468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"client":"2","":"downstairs","session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.3350568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335065909Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"fb589f7e-f94b-4791-a26c-170868f716ec is now active with session: 74d473fc-2c4c-4997-a133-dfb307060498","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335075738Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335085557Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"session_id":"74d473fc-2c4c-4997-a133-dfb307060498"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335094396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Activated sub_volume fb589f7e-f94b-4791-a26c-170868f716ec","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335102136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335110035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.335413419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} 01/30 Write block 94 len 20480 data: 7 5 3 5 6 02/30 Read block 245 len 24576 03/30 Read block 341 len 8192 04/30 Write block 463 len 28672 data: 12 11 10 8 7 5 8 05/30 Write block 580 len 4096 data: 4 06/30 Write block 192 len 4096 data: 9 07/30 Write block 247 len 8192 data: 12 12 08/30 Read block 217 len 28672 09/30 Read block 328 len 36864 10/30 Write block 110 len 8192 data: 9 7 11/30 Flush 12/30 Write block 284 len 24576 data: 8 8 6 6 6 5 13/30 Write block 218 len 20480 data: 11 13 11 9 6 14/30 Read block 479 len 4096 15/30 Write block 514 len 16384 data: 13 12 13 12 16/30 Write block 232 len 36864 data: 10 9 10 13 11 12 12 10 10 17/30 Read block 209 len 4096 18/30 Read block 37 len 12288 19/30 Read block 41 len 4096 20/30 Read block 349 len 8192 21/30 Write block 228 len 36864 data: 11 10 11 11 11 10 11 14 12 22/30 Read block 254 len 20480 23/30 Read block 106 len 16384 24/30 Read block 102 len 32768 25/30 Write block 81 len 36864 data: 9 7 7 7 8 10 8 12 12 26/30 Read block 472 len 32768 27/30 Read block 456 len 12288 28/30 Read block 232 len 32768 29/30 Write block 77 len 32768 data: 6 7 7 9 10 8 8 8 30/30 Write block 485 len 24576 data: 8 10 11 10 9 9 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 2 28 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:30.343457258Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:30.343630793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1627,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 120 120 120 194 194 194 F F F 2 040-059 120 120 120 194 194 194 F F F 3 060-079 121 121 121 196 196 196 F T T <--- 4 080-099 121 122 122 196 197 197 T T T <--- 5 100-119 112 122 122 179 197 197 F F F <--- 6 120-139 120 120 120 195 195 195 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 121 121 121 196 196 196 F F F 9 180-199 121 122 122 196 197 197 F F F <--- 10 200-219 120 120 120 195 195 195 F T T <--- 11 220-239 120 120 120 195 195 195 F T T <--- 12 240-259 112 122 122 179 197 197 F T T <--- 13 260-279 121 121 121 196 196 196 F F F 14 280-299 118 118 118 190 190 190 F T T <--- 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 120 120 120 194 194 194 F F F 20 400-419 119 119 119 193 193 193 F F F 21 420-439 121 121 121 196 196 196 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 118 122 122 191 197 197 F F F <--- 24 480-499 119 119 119 193 193 193 F T T <--- 25 500-519 119 119 119 193 193 193 F T T <--- 26 520-539 116 116 116 188 188 188 F F F 27 540-559 121 121 121 196 196 196 F F F 28 560-579 121 121 121 196 196 196 F F F 29 580-599 113 122 122 181 197 197 F F F <--- Max gen: 122, Max flush: 197 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 120 120 194 194 194 F F F 1 020-039 120 120 120 194 194 194 F F F 2 040-059 120 120 120 194 194 194 F F F 3 060-079 123 123 123 198 198 198 F F F 4 080-099 123 123 123 198 198 198 F F F 5 100-119 122 122 122 197 197 197 F F F 6 120-139 120 120 120 195 195 195 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 121 121 121 196 196 196 F F F 9 180-199 122 122 122 197 197 197 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 123 123 123 198 198 198 F F F 12 240-259 123 123 123 198 198 198 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 123 123 123 198 198 198 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 117 117 189 189 189 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 120 120 120 194 194 194 F F F 20 400-419 119 119 119 193 193 193 F F F 21 420-439 121 121 121 196 196 196 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 122 122 122 197 197 197 F F F 24 480-499 123 123 123 198 198 198 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 116 116 116 188 188 188 F F F 27 540-559 121 121 121 196 196 196 F F F 28 560-579 121 121 121 196 196 196 F F F 29 580-599 122 122 122 197 197 197 F F F Max gen: 123, Max flush: 198 Begin loop 62 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.602195293Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:34.609375788Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.609424214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.616495508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.616743026Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.616770704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.616779053Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.616790482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Crucible 83c05427-a849-4b18-909b-b569ae538b1b has session id: 6aec93af-ad82-48e2-af05-80844eba101a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617113954Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Upstairs opts: Upstairs UUID: 83c05427-a849-4b18-909b-b569ae538b1b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617138722Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Crucible stats registered with UUID: 83c05427-a849-4b18-909b-b569ae538b1b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617150651Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:34.61715957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617311626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617343014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617353553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617478912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617491051Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"83c05427-a849-4b18-909b-b569ae538b1b active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617514799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617523848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617533327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617542746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617552086Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617561935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617571624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41205)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617632059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"ds_connection connected from Ok(127.0.0.1:54622)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617642808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"ds_connection connected from Ok(127.0.0.1:62109)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.617654897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618297941Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618323248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618377014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618611763Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.6186448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[0]R flush_numbers[0..12]: [194, 194, 194, 198, 198, 197, 195, 191, 196, 197, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618694646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[0]R generation[0..12]: [120, 120, 120, 123, 123, 122, 120, 118, 121, 122, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618705665Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618716594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[1]R flush_numbers[0..12]: [194, 194, 194, 198, 198, 197, 195, 191, 196, 197, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618727083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[1]R generation[0..12]: [120, 120, 120, 123, 123, 122, 120, 118, 121, 122, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618737382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618748071Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[2]R flush_numbers[0..12]: [194, 194, 194, 198, 198, 197, 195, 191, 196, 197, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.61875846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[2]R generation[0..12]: [120, 120, 120, 123, 123, 122, 120, 118, 121, 122, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618771039Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618781818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618791058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618800667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Next flush: 199","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618809806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618819025Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"":"downstairs","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618828624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618837264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"0","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618846733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"1","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618856222Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"client":"2","":"downstairs","session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618865721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.61887456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"83c05427-a849-4b18-909b-b569ae538b1b is now active with session: 6aec93af-ad82-48e2-af05-80844eba101a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618884329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618892919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"session_id":"6aec93af-ad82-48e2-af05-80844eba101a"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618901798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Activated sub_volume 83c05427-a849-4b18-909b-b569ae538b1b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618909607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.618917537Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.619282345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} 01/30 Write block 18 len 12288 data: 8 9 7 02/30 Write block 174 len 16384 data: 6 5 7 7 03/30 Read block 478 len 8192 04/30 Read block 95 len 8192 05/30 Write block 558 len 4096 data: 8 06/30 Read block 589 len 32768 07/30 Read block 15 len 32768 08/30 Read block 75 len 8192 09/30 Flush 10/30 Write block 342 len 12288 data: 10 11 10 11/30 Read block 45 len 28672 12/30 Write block 166 len 8192 data: 5 5 13/30 Flush 14/30 Write block 428 len 32768 data: 10 10 11 10 10 9 9 8 15/30 Write block 176 len 20480 data: 8 8 5 6 6 16/30 Read block 204 len 8192 17/30 Flush 18/30 Write block 282 len 24576 data: 9 8 9 9 7 7 19/30 Flush 20/30 Write block 468 len 28672 data: 6 9 9 7 7 8 13 21/30 Read block 317 len 12288 22/30 Flush 23/30 Write block 577 len 4096 data: 5 24/30 Read block 429 len 28672 25/30 Write block 527 len 20480 data: 10 9 9 8 9 26/30 Read block 495 len 36864 27/30 Read block 233 len 12288 28/30 Write block 415 len 28672 data: 10 15 13 15 12 13 11 29/30 Write block 386 len 20480 data: 14 17 16 16 15 30/30 Read block 389 len 36864 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:34.628101916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:34.628279771Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1635,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 120 124 124 194 199 199 F F F <--- 1 020-039 120 124 124 194 199 199 F F F <--- 2 040-059 120 120 120 194 194 194 F F F 3 060-079 123 123 123 198 198 198 F F F 4 080-099 123 123 123 198 198 198 F F F 5 100-119 122 122 122 197 197 197 F F F 6 120-139 120 120 120 195 195 195 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 121 124 124 196 201 201 F F F <--- 9 180-199 122 124 124 197 201 201 F F F <--- 10 200-219 123 123 123 198 198 198 F F F 11 220-239 123 123 123 198 198 198 F F F 12 240-259 123 123 123 198 198 198 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 123 124 124 198 202 202 F F F <--- 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 117 124 124 189 200 200 F F F <--- 18 360-379 118 118 118 191 191 191 F F F 19 380-399 120 120 120 194 194 194 F T T <--- 20 400-419 119 119 119 193 193 193 F T T <--- 21 420-439 121 124 124 196 201 201 F T T <--- 22 440-459 118 118 118 190 190 190 F F F 23 460-479 122 124 124 197 203 203 F F F <--- 24 480-499 123 123 123 198 198 198 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 116 116 116 188 188 188 F T T <--- 27 540-559 121 124 124 196 199 199 F F F <--- 28 560-579 121 121 121 196 196 196 F T T <--- 29 580-599 122 122 122 197 197 197 F F F Max gen: 124, Max flush: 203 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 124 199 199 199 F F F 1 020-039 124 124 124 199 199 199 F F F 2 040-059 120 120 120 194 194 194 F F F 3 060-079 123 123 123 198 198 198 F F F 4 080-099 123 123 123 198 198 198 F F F 5 100-119 122 122 122 197 197 197 F F F 6 120-139 120 120 120 195 195 195 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 124 124 124 201 201 201 F F F 9 180-199 124 124 124 201 201 201 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 123 123 123 198 198 198 F F F 12 240-259 123 123 123 198 198 198 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 124 124 124 202 202 202 F F F 15 300-319 118 118 118 190 190 190 F F F 16 320-339 110 110 110 174 174 174 F F F 17 340-359 124 124 124 200 200 200 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 125 125 125 204 204 204 F F F 20 400-419 125 125 125 204 204 204 F F F 21 420-439 125 125 125 204 204 204 F F F 22 440-459 118 118 118 190 190 190 F F F 23 460-479 124 124 124 203 203 203 F F F 24 480-499 123 123 123 198 198 198 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 125 125 125 204 204 204 F F F 27 540-559 124 124 124 199 199 199 F F F 28 560-579 125 125 125 204 204 204 F F F 29 580-599 122 122 122 197 197 197 F F F Max gen: 125, Max flush: 204 Begin loop 63 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.891275686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.899621649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.899910813Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.899936191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.89994471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.899956329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Crucible fb126619-e730-45fc-b005-fdd5f7304d79 has session id: 9a6445d7-54c8-4047-b9a1-0b971d5c4037","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900315588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Upstairs opts: Upstairs UUID: fb126619-e730-45fc-b005-fdd5f7304d79, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900343966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Crucible stats registered with UUID: fb126619-e730-45fc-b005-fdd5f7304d79","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900354935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:38.900363934Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900484773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900501322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900511151Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900665038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900688746Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"fb126619-e730-45fc-b005-fdd5f7304d79 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900712214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900721403Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900730992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900740511Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.9007499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900759889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900769569Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"ds_connection connected from Ok(127.0.0.1:39701)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900804016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"ds_connection connected from Ok(127.0.0.1:43159)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900814685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"ds_connection connected from Ok(127.0.0.1:33194)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.900888428Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901423782Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.90144285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901526723Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901736004Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901747483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[0]R flush_numbers[0..12]: [199, 199, 194, 198, 198, 197, 195, 191, 201, 201, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901803878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[0]R generation[0..12]: [124, 124, 120, 123, 123, 122, 120, 118, 124, 124, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901814817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901825547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[1]R flush_numbers[0..12]: [199, 199, 194, 198, 198, 197, 195, 191, 201, 201, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901835976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[1]R generation[0..12]: [124, 124, 120, 123, 123, 122, 120, 118, 124, 124, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901846135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901856734Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[2]R flush_numbers[0..12]: [199, 199, 194, 198, 198, 197, 195, 191, 201, 201, 198, 198]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901867213Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[2]R generation[0..12]: [124, 124, 120, 123, 123, 122, 120, 118, 124, 124, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901877482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901888181Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.90189728Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901906889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Next flush: 205","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901915869Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901924978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"":"downstairs","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901935607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901944096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"0","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901953575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"1","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901963115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"client":"2","":"downstairs","session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901972774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901981593Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"fb126619-e730-45fc-b005-fdd5f7304d79 is now active with session: 9a6445d7-54c8-4047-b9a1-0b971d5c4037","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.901991162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.90201074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"session_id":"9a6445d7-54c8-4047-b9a1-0b971d5c4037"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.9020196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Activated sub_volume fb126619-e730-45fc-b005-fdd5f7304d79","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.902027539Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.902035378Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.902379418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} 01/30 Write block 23 len 24576 data: 6 8 8 8 8 8 02/30 Flush 03/30 Write block 74 len 16384 data: 6 7 6 7 04/30 Read block 565 len 8192 05/30 Read block 337 len 36864 06/30 Read block 126 len 40960 07/30 Read block 355 len 8192 08/30 Flush 09/30 Write block 131 len 28672 data: 14 13 11 11 11 12 11 10/30 Write block 181 len 20480 data: 4 4 7 8 8 11/30 Write block 83 len 40960 data: 9 9 9 11 9 13 13 10 7 8 12/30 Read block 202 len 24576 13/30 Write block 381 len 36864 data: 13 11 14 12 11 15 18 17 17 14/30 Read block 168 len 32768 15/30 Read block 505 len 32768 16/30 Write block 58 len 20480 data: 7 9 8 11 7 17/30 Read block 586 len 36864 18/30 Flush 19/30 Write block 100 len 28672 data: 7 8 8 10 11 10 8 20/30 Read block 40 len 32768 21/30 Write block 228 len 32768 data: 12 11 12 12 12 11 12 15 22/30 Write block 128 len 16384 data: 17 16 14 15 23/30 Write block 458 len 36864 data: 12 10 10 9 10 13 12 11 9 24/30 Read block 43 len 20480 25/30 Write block 380 len 16384 data: 12 14 12 15 26/30 Flush 27/30 Write block 2 len 40960 data: 6 7 6 7 8 9 10 12 10 10 28/30 Write block 318 len 24576 data: 10 13 13 11 13 13 29/30 Read block 360 len 32768 30/30 Read block 371 len 8192 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 4 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1001)) Some(JobId(1025)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:38.911813506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:38.911983342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1643,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 124 199 199 199 T F T <--- 1 020-039 126 126 126 205 205 205 F F F 2 040-059 126 120 126 207 194 207 F F F <--- 3 060-079 126 123 126 207 198 207 F T F <--- 4 080-099 126 123 126 207 198 207 F F F <--- 5 100-119 126 122 126 208 197 208 F F F <--- 6 120-139 126 120 126 208 195 208 F F F <--- 7 140-159 118 118 118 191 191 191 F F F 8 160-179 124 124 124 201 201 201 F F F 9 180-199 126 124 126 207 201 207 F F F <--- 10 200-219 123 123 123 198 198 198 F F F 11 220-239 126 123 126 208 198 208 F F F <--- 12 240-259 123 123 123 198 198 198 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 124 124 124 202 202 202 F F F 15 300-319 118 118 118 190 190 190 T F T <--- 16 320-339 110 110 110 174 174 174 T F T <--- 17 340-359 124 124 124 200 200 200 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 126 125 126 208 204 208 F F F <--- 20 400-419 125 125 125 204 204 204 F F F 21 420-439 125 125 125 204 204 204 F F F 22 440-459 126 118 126 208 190 208 F F F <--- 23 460-479 126 124 126 208 203 208 F F F <--- 24 480-499 123 123 123 198 198 198 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 125 125 125 204 204 204 F F F 27 540-559 124 124 124 199 199 199 F F F 28 560-579 125 125 125 204 204 204 F F F 29 580-599 122 122 122 197 197 197 F F F Max gen: 126, Max flush: 208 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 126 126 126 205 205 205 F F F 2 040-059 126 126 126 207 207 207 F F F 3 060-079 126 126 126 207 207 207 F F F 4 080-099 126 126 126 207 207 207 F F F 5 100-119 126 126 126 208 208 208 F F F 6 120-139 126 126 126 208 208 208 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 124 124 124 201 201 201 F F F 9 180-199 126 126 126 207 207 207 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 126 126 126 208 208 208 F F F 12 240-259 123 123 123 198 198 198 F F F 13 260-279 121 121 121 196 196 196 F F F 14 280-299 124 124 124 202 202 202 F F F 15 300-319 127 127 127 209 209 209 F F F 16 320-339 127 127 127 209 209 209 F F F 17 340-359 124 124 124 200 200 200 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 126 126 126 208 208 208 F F F 20 400-419 125 125 125 204 204 204 F F F 21 420-439 125 125 125 204 204 204 F F F 22 440-459 126 126 126 208 208 208 F F F 23 460-479 126 126 126 208 208 208 F F F 24 480-499 123 123 123 198 198 198 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 125 125 125 204 204 204 F F F 27 540-559 124 124 124 199 199 199 F F F 28 560-579 125 125 125 204 204 204 F F F 29 580-599 122 122 122 197 197 197 F F F Max gen: 127, Max flush: 209 Begin loop 64 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.165939614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174218743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174507758Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174533665Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174544575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174556054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Crucible f2cfb88c-832a-4161-a8ea-a3e8da38e996 has session id: 70ba0f69-f6d9-46f6-a80c-37a45f73b907","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174847708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Upstairs opts: Upstairs UUID: f2cfb88c-832a-4161-a8ea-a3e8da38e996, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174878525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Crucible stats registered with UUID: f2cfb88c-832a-4161-a8ea-a3e8da38e996","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.174889355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:43.174898414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175044691Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175074628Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175085207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175241744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175256952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"f2cfb88c-832a-4161-a8ea-a3e8da38e996 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175291749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175300999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175310538Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175319957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175331396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175341225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175350824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"ds_connection connected from Ok(127.0.0.1:63918)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.17540159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"ds_connection connected from Ok(127.0.0.1:52760)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175425768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"ds_connection connected from Ok(127.0.0.1:55661)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.175488382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176025356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176064882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176124817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176353247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176372585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[0]R flush_numbers[0..12]: [209, 205, 207, 207, 207, 208, 208, 191, 201, 207, 198, 208]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176460478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[0]R generation[0..12]: [127, 126, 126, 126, 126, 126, 126, 118, 124, 126, 123, 126]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176471597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176482536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[1]R flush_numbers[0..12]: [209, 205, 207, 207, 207, 208, 208, 191, 201, 207, 198, 208]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176493105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[1]R generation[0..12]: [127, 126, 126, 126, 126, 126, 126, 118, 124, 126, 123, 126]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176506154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176517053Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[2]R flush_numbers[0..12]: [209, 205, 207, 207, 207, 208, 208, 191, 201, 207, 198, 208]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176527582Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[2]R generation[0..12]: [127, 126, 126, 126, 126, 126, 126, 118, 124, 126, 123, 126]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176537941Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.17654866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176558149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176567968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Next flush: 210","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176577257Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176586467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"":"downstairs","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176596256Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176604945Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"0","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176614564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"1","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176624243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"client":"2","":"downstairs","session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176634043Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176642962Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"f2cfb88c-832a-4161-a8ea-a3e8da38e996 is now active with session: 70ba0f69-f6d9-46f6-a80c-37a45f73b907","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176653961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.1766625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"session_id":"70ba0f69-f6d9-46f6-a80c-37a45f73b907"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176671769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Activated sub_volume f2cfb88c-832a-4161-a8ea-a3e8da38e996","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176679689Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.176687618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.177059325Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} 01/30 Write block 468 len 12288 data: 7 10 10 02/30 Write block 234 len 32768 data: 13 16 13 13 13 11 11 8 03/30 Read block 417 len 32768 04/30 Write block 443 len 4096 data: 12 05/30 Read block 208 len 40960 06/30 Read block 311 len 24576 07/30 Write block 260 len 8192 data: 11 15 08/30 Write block 466 len 32768 data: 10 8 8 11 11 8 8 9 09/30 Write block 579 len 40960 data: 3 5 4 6 6 10 11 14 12 13 10/30 Read block 250 len 20480 11/30 Flush 12/30 Write block 405 len 24576 data: 14 14 13 12 11 11 13/30 Read block 188 len 8192 14/30 Write block 479 len 32768 data: 11 9 11 12 12 11 9 11 15/30 Flush 16/30 Write block 99 len 36864 data: 7 8 9 9 11 12 11 9 9 17/30 Write block 175 len 16384 data: 6 9 9 6 18/30 Flush 19/30 Write block 336 len 28672 data: 11 13 15 14 12 12 11 20/30 Write block 482 len 12288 data: 13 13 12 21/30 Read block 386 len 36864 22/30 Write block 28 len 16384 data: 9 6 6 9 23/30 Write block 50 len 28672 data: 12 11 10 10 12 10 8 24/30 Flush 25/30 Write block 570 len 40960 data: 6 8 9 8 8 8 8 6 4 4 26/30 Read block 52 len 20480 27/30 Read block 14 len 4096 28/30 Write block 290 len 40960 data: 6 8 8 11 10 12 13 12 12 15 29/30 Read block 254 len 8192 30/30 Write block 532 len 28672 data: 9 9 12 11 11 11 9 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Write 3 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 19 1 21 Done 18 0 18 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1010)) Some(JobId(1023)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:43.186370814Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:43.186567577Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1651,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 128 126 128 213 205 213 F F F <--- 2 040-059 128 126 128 213 207 213 F F F <--- 3 060-079 126 126 126 207 207 207 F F F 4 080-099 128 126 128 212 207 212 F F F <--- 5 100-119 128 126 128 212 208 212 F F F <--- 6 120-139 126 126 126 208 208 208 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 128 124 128 212 201 212 F F F <--- 9 180-199 126 126 126 207 207 207 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 128 128 128 210 210 210 F F F 12 240-259 128 128 128 210 210 210 F F F 13 260-279 128 128 128 210 210 210 F F F 14 280-299 124 124 124 202 202 202 T F T <--- 15 300-319 127 127 127 209 209 209 F F F 16 320-339 128 127 128 213 209 213 F F F <--- 17 340-359 128 124 128 213 200 213 F F F <--- 18 360-379 118 118 118 191 191 191 F F F 19 380-399 126 126 126 208 208 208 F F F 20 400-419 128 125 128 211 204 211 F F F <--- 21 420-439 125 125 125 204 204 204 F F F 22 440-459 128 128 128 210 210 210 F F F 23 460-479 128 128 128 211 210 211 F F F <--- 24 480-499 128 123 128 213 198 213 F F F <--- 25 500-519 123 123 123 198 198 198 F F F 26 520-539 125 125 125 204 204 204 T F T <--- 27 540-559 124 124 124 199 199 199 F F F 28 560-579 128 128 128 210 210 210 T F T <--- 29 580-599 128 128 128 210 210 210 F F F Max gen: 128, Max flush: 213 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 128 128 128 213 213 213 F F F 2 040-059 128 128 128 213 213 213 F F F 3 060-079 126 126 126 207 207 207 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 F F F 6 120-139 126 126 126 208 208 208 F F F 7 140-159 118 118 118 191 191 191 F F F 8 160-179 128 128 128 212 212 212 F F F 9 180-199 126 126 126 207 207 207 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 128 128 128 210 210 210 F F F 12 240-259 128 128 128 210 210 210 F F F 13 260-279 128 128 128 210 210 210 F F F 14 280-299 129 129 129 214 214 214 F F F 15 300-319 127 127 127 209 209 209 F F F 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 F F F 18 360-379 118 118 118 191 191 191 F F F 19 380-399 126 126 126 208 208 208 F F F 20 400-419 128 128 128 211 211 211 F F F 21 420-439 125 125 125 204 204 204 F F F 22 440-459 128 128 128 210 210 210 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 F F F 25 500-519 123 123 123 198 198 198 F F F 26 520-539 129 129 129 214 214 214 F F F 27 540-559 124 124 124 199 199 199 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 128 128 128 210 210 210 F F F Max gen: 129, Max flush: 214 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.437714684Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:47.444896239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.444946264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45200226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452251138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452275166Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452283655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452294984Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Crucible 422e1acc-f6a5-4ae1-b14e-39e06fcccdb3 has session id: bf8cd606-d147-403e-bebc-556e7f3d9a8e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45268772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Upstairs opts: Upstairs UUID: 422e1acc-f6a5-4ae1-b14e-39e06fcccdb3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452717957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Crucible stats registered with UUID: 422e1acc-f6a5-4ae1-b14e-39e06fcccdb3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452730916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:47.452740075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452884843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45291313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.452923999Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453077206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453094075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"422e1acc-f6a5-4ae1-b14e-39e06fcccdb3 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453109963Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453118752Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453128322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453137741Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45314705Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453156759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453166338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:53014)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453238762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:42498)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45326476Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:37323)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453278079Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453878766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453891905Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.453996616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454204668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454213947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[0]R flush_numbers[0..12]: [209, 213, 213, 207, 212, 212, 208, 191, 212, 207, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454278791Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[0]R generation[0..12]: [127, 128, 128, 126, 128, 128, 126, 118, 128, 126, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45428974Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454300579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[1]R flush_numbers[0..12]: [209, 213, 213, 207, 212, 212, 208, 191, 212, 207, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454311059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[1]R generation[0..12]: [127, 128, 128, 126, 128, 128, 126, 118, 128, 126, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454321388Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454332077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[2]R flush_numbers[0..12]: [209, 213, 213, 207, 212, 212, 208, 191, 212, 207, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454342436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[2]R generation[0..12]: [127, 128, 128, 126, 128, 128, 126, 118, 128, 126, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454352715Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454365034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454374233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454383742Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Next flush: 215","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454392841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454401921Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"":"downstairs","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.45441134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454419829Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"0","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454429528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"1","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454439267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"client":"2","":"downstairs","session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454449107Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454458206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"422e1acc-f6a5-4ae1-b14e-39e06fcccdb3 is now active with session: bf8cd606-d147-403e-bebc-556e7f3d9a8e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454467945Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454476534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"session_id":"bf8cd606-d147-403e-bebc-556e7f3d9a8e"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454485453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Activated sub_volume 422e1acc-f6a5-4ae1-b14e-39e06fcccdb3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454493253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454501132Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.454891208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} 01/30 Write block 517 len 36864 data: 13 12 13 12 12 12 10 12 10 02/30 Read block 475 len 20480 03/30 Read block 165 len 12288 04/30 Write block 367 len 20480 data: 12 14 13 9 10 05/30 Read block 438 len 32768 06/30 Write block 508 len 8192 data: 11 10 07/30 Write block 125 len 8192 data: 9 10 08/30 Read block 63 len 16384 09/30 Write block 419 len 20480 data: 13 14 12 10 9 10/30 Read block 588 len 28672 11/30 Read block 462 len 28672 12/30 Write block 368 len 4096 data: 15 13/30 Write block 190 len 8192 data: 8 9 14/30 Write block 435 len 32768 data: 9 8 9 11 10 12 12 14 15/30 Read block 102 len 20480 16/30 Read block 269 len 24576 17/30 Flush 18/30 Write block 392 len 40960 data: 21 15 16 15 11 10 13 13 15 17 19/30 Write block 446 len 12288 data: 14 12 12 20/30 Write block 424 len 36864 data: 8 7 7 7 11 11 12 11 11 21/30 Write block 444 len 24576 data: 17 14 15 13 13 12 22/30 Read block 403 len 36864 23/30 Read block 130 len 8192 24/30 Read block 293 len 4096 25/30 Write block 592 len 4096 data: 15 26/30 Write block 24 len 12288 data: 9 9 9 27/30 Write block 153 len 24576 data: 14 14 14 12 13 12 28/30 Read block 253 len 36864 29/30 Read block 224 len 28672 30/30 Read block 38 len 4096 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:47.462744304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:47.46290379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1659,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 128 128 128 213 213 213 F T T <--- 2 040-059 128 128 128 213 213 213 F F F 3 060-079 126 126 126 207 207 207 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 F F F 6 120-139 126 130 130 208 215 215 F F F <--- 7 140-159 118 118 118 191 191 191 F T T <--- 8 160-179 128 128 128 212 212 212 F F F 9 180-199 126 130 130 207 215 215 F F F <--- 10 200-219 123 123 123 198 198 198 F F F 11 220-239 128 128 128 210 210 210 F F F 12 240-259 128 128 128 210 210 210 F F F 13 260-279 128 128 128 210 210 210 F F F 14 280-299 129 129 129 214 214 214 F F F 15 300-319 127 127 127 209 209 209 F F F 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 F F F 18 360-379 118 130 130 191 215 215 F F F <--- 19 380-399 126 126 126 208 208 208 F T T <--- 20 400-419 128 130 130 211 215 215 F T T <--- 21 420-439 125 130 130 204 215 215 F T T <--- 22 440-459 128 130 130 210 215 215 F T T <--- 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 F F F 25 500-519 123 130 130 198 215 215 T F F <--- 26 520-539 129 130 130 214 215 215 T F F <--- 27 540-559 124 124 124 199 199 199 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 128 128 128 210 210 210 F T T <--- Max gen: 130, Max flush: 215 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 131 131 131 216 216 216 F F F 2 040-059 128 128 128 213 213 213 F F F 3 060-079 126 126 126 207 207 207 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 F F F 6 120-139 130 130 130 215 215 215 F F F 7 140-159 131 131 131 216 216 216 F F F 8 160-179 128 128 128 212 212 212 F F F 9 180-199 130 130 130 215 215 215 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 128 128 128 210 210 210 F F F 12 240-259 128 128 128 210 210 210 F F F 13 260-279 128 128 128 210 210 210 F F F 14 280-299 129 129 129 214 214 214 F F F 15 300-319 127 127 127 209 209 209 F F F 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 F F F 18 360-379 130 130 130 215 215 215 F F F 19 380-399 131 131 131 216 216 216 F F F 20 400-419 131 131 131 216 216 216 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 F F F 25 500-519 130 130 130 215 215 215 F F F 26 520-539 130 130 130 215 215 215 F F F 27 540-559 124 124 124 199 199 199 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 131 131 131 216 216 216 F F F Max gen: 131, Max flush: 216 Begin loop 66 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.721757135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:51.728920631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.728970246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.735830739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736088536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736113594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736121883Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736133162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Crucible 7956c463-a45f-41d7-ac42-4609942707a7 has session id: 5d8e9201-02d0-4ca4-836d-eae1534c79b4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736454464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Upstairs opts: Upstairs UUID: 7956c463-a45f-41d7-ac42-4609942707a7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736482802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Crucible stats registered with UUID: 7956c463-a45f-41d7-ac42-4609942707a7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736493501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:51.73650259Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736630599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736649897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736659897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736802884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736826502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"7956c463-a45f-41d7-ac42-4609942707a7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.73685171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736860819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736870388Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736879767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736889037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736898806Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736908315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:65197)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.736957011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:32802)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.73707765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56868)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.737103288Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.737669399Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.737686447Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.737704366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738011499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738054765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[0]R flush_numbers[0..12]: [209, 216, 213, 207, 212, 212, 215, 216, 212, 215, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738101521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[0]R generation[0..12]: [127, 131, 128, 126, 128, 128, 130, 131, 128, 130, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.73811249Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738123309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[1]R flush_numbers[0..12]: [209, 216, 213, 207, 212, 212, 215, 216, 212, 215, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738133718Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[1]R generation[0..12]: [127, 131, 128, 126, 128, 128, 130, 131, 128, 130, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738144067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738154936Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[2]R flush_numbers[0..12]: [209, 216, 213, 207, 212, 212, 215, 216, 212, 215, 198, 210]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738165335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[2]R generation[0..12]: [127, 131, 128, 126, 128, 128, 130, 131, 128, 130, 123, 128]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738175714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738186464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738195613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738205012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Next flush: 217","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738215981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.73822519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"":"downstairs","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738234869Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738243499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"0","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738252878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"1","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738262197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"client":"2","":"downstairs","session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738271896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738280875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"7956c463-a45f-41d7-ac42-4609942707a7 is now active with session: 5d8e9201-02d0-4ca4-836d-eae1534c79b4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738290584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738299574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738307073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"session_id":"5d8e9201-02d0-4ca4-836d-eae1534c79b4"} {"msg":"Activated sub_volume 7956c463-a45f-41d7-ac42-4609942707a7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738316042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738323992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.738650243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} 01/30 Write block 288 len 40960 data: 7 6 7 9 9 12 11 13 14 13 02/30 Read block 346 len 24576 03/30 Write block 286 len 16384 data: 8 8 8 7 04/30 Write block 231 len 40960 data: 13 13 12 14 17 14 14 14 12 12 05/30 Write block 405 len 16384 data: 15 15 14 13 06/30 Read block 138 len 36864 07/30 Read block 426 len 40960 08/30 Write block 286 len 32768 data: 9 9 9 8 8 10 10 13 09/30 Read block 365 len 36864 10/30 Write block 305 len 40960 data: 12 11 11 11 11 12 9 9 8 9 11/30 Write block 146 len 4096 data: 10 12/30 Read block 444 len 16384 13/30 Read block 186 len 40960 14/30 Read block 338 len 24576 15/30 Write block 61 len 24576 data: 12 8 10 9 9 9 16/30 Write block 554 len 24576 data: 7 11 9 8 9 9 17/30 Read block 315 len 8192 18/30 Flush 19/30 Write block 259 len 32768 data: 12 12 16 13 11 12 14 15 20/30 Write block 531 len 28672 data: 10 10 10 13 12 12 12 21/30 Flush 22/30 Write block 302 len 32768 data: 13 15 15 13 12 12 12 12 23/30 Flush 24/30 Write block 394 len 16384 data: 17 16 12 11 25/30 Read block 3 len 12288 26/30 Write block 385 len 40960 data: 12 16 19 18 18 16 16 22 16 18 27/30 Read block 47 len 32768 28/30 Read block 246 len 16384 29/30 Read block 147 len 16384 30/30 Write block 520 len 20480 data: 13 13 13 11 13 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:51.748106569Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:51.748280644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1667,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 131 131 131 216 216 216 F F F 2 040-059 128 128 128 213 213 213 F F F 3 060-079 126 132 132 207 217 217 F F F <--- 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 F F F 6 120-139 130 130 130 215 215 215 F F F 7 140-159 131 132 132 216 217 217 F F F <--- 8 160-179 128 128 128 212 212 212 F F F 9 180-199 130 130 130 215 215 215 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 128 132 132 210 217 217 F F F <--- 12 240-259 128 132 132 210 218 218 F F F <--- 13 260-279 128 132 132 210 218 218 F F F <--- 14 280-299 129 132 132 214 217 217 F F F <--- 15 300-319 127 132 132 209 219 219 F F F <--- 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 F F F 18 360-379 130 130 130 215 215 215 F F F 19 380-399 131 131 131 216 216 216 F T T <--- 20 400-419 131 132 132 216 217 217 F F F <--- 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 F F F 25 500-519 130 130 130 215 215 215 F F F 26 520-539 130 132 132 215 218 218 F T T <--- 27 540-559 124 132 132 199 217 217 F F F <--- 28 560-579 129 129 129 214 214 214 F F F 29 580-599 131 131 131 216 216 216 F F F Max gen: 132, Max flush: 219 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 131 131 131 216 216 216 F F F 2 040-059 128 128 128 213 213 213 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 F F F 6 120-139 130 130 130 215 215 215 F F F 7 140-159 132 132 132 217 217 217 F F F 8 160-179 128 128 128 212 212 212 F F F 9 180-199 130 130 130 215 215 215 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 132 132 132 217 217 217 F F F 12 240-259 132 132 132 218 218 218 F F F 13 260-279 132 132 132 218 218 218 F F F 14 280-299 132 132 132 217 217 217 F F F 15 300-319 132 132 132 219 219 219 F F F 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 F F F 18 360-379 130 130 130 215 215 215 F F F 19 380-399 133 133 133 220 220 220 F F F 20 400-419 132 132 132 217 217 217 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 F F F 25 500-519 130 130 130 215 215 215 F F F 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 131 131 131 216 216 216 F F F Max gen: 133, Max flush: 220 Begin loop 67 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.002625553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.010994124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011279749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011304906Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011313066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011324325Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Crucible 4e72314a-64b4-4845-b45d-d65d25d8fe8d has session id: 609fd937-397f-4be9-9c57-6d27985f9206","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011654206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Upstairs opts: Upstairs UUID: 4e72314a-64b4-4845-b45d-d65d25d8fe8d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011730289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Crucible stats registered with UUID: 4e72314a-64b4-4845-b45d-d65d25d8fe8d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011755667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:56.011765716Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011883406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011913223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.011923953Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012104457Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012122005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"4e72314a-64b4-4845-b45d-d65d25d8fe8d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012145173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012154332Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012166031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012175521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.0121848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012194899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012204578Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"ds_connection connected from Ok(127.0.0.1:37361)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012241915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"ds_connection connected from Ok(127.0.0.1:47730)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012267863Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"ds_connection connected from Ok(127.0.0.1:63708)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012279522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012948873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.012968442Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013014987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013233028Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013266796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[0]R flush_numbers[0..12]: [209, 216, 213, 217, 212, 212, 215, 217, 212, 215, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013320311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[0]R generation[0..12]: [127, 131, 128, 132, 128, 128, 130, 132, 128, 130, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.01333117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013344179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[1]R flush_numbers[0..12]: [209, 216, 213, 217, 212, 212, 215, 217, 212, 215, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013354548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[1]R generation[0..12]: [127, 131, 128, 132, 128, 128, 130, 132, 128, 130, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013365047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013375766Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[2]R flush_numbers[0..12]: [209, 216, 213, 217, 212, 212, 215, 217, 212, 215, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013386215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[2]R generation[0..12]: [127, 131, 128, 132, 128, 128, 130, 132, 128, 130, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013396464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013407303Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013416622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013426202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Next flush: 221","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013435441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.01344471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"":"downstairs","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013454329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013463028Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"0","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013472818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"1","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013482447Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"client":"2","":"downstairs","session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013493236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013502255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"4e72314a-64b4-4845-b45d-d65d25d8fe8d is now active with session: 609fd937-397f-4be9-9c57-6d27985f9206","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013511994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013520583Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"session_id":"609fd937-397f-4be9-9c57-6d27985f9206"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013529473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Activated sub_volume 4e72314a-64b4-4845-b45d-d65d25d8fe8d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013537352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013545271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.013862504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} 01/30 Write block 596 len 16384 data: 6 6 5 3 02/30 Write block 399 len 12288 data: 14 16 18 03/30 Read block 268 len 12288 04/30 Read block 196 len 36864 05/30 Read block 537 len 12288 06/30 Read block 533 len 24576 07/30 Read block 97 len 20480 08/30 Write block 595 len 8192 data: 7 7 09/30 Flush 10/30 Write block 514 len 4096 data: 14 11/30 Write block 305 len 4096 data: 14 12/30 Read block 148 len 28672 13/30 Read block 220 len 16384 14/30 Read block 374 len 4096 15/30 Write block 35 len 12288 data: 11 9 9 16/30 Read block 263 len 36864 17/30 Read block 288 len 8192 18/30 Flush 19/30 Write block 131 len 4096 data: 16 20/30 Read block 546 len 16384 21/30 Write block 358 len 28672 data: 8 8 7 12 12 13 12 22/30 Write block 498 len 8192 data: 11 12 23/30 Write block 178 len 40960 data: 7 7 7 5 5 8 9 9 7 7 24/30 Read block 570 len 8192 25/30 Read block 540 len 36864 26/30 Read block 170 len 40960 27/30 Write block 340 len 32768 data: 13 13 12 12 11 12 10 8 28/30 Read block 257 len 20480 29/30 Write block 115 len 36864 data: 7 8 7 9 8 9 8 10 10 30/30 Read block 523 len 16384 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Write 3 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:32:56.022251473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:32:56.022432537Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1675,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 134 131 134 222 216 222 F F F <--- 2 040-059 128 128 128 213 213 213 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 128 128 128 212 212 212 T F T <--- 6 120-139 130 130 130 215 215 215 T F T <--- 7 140-159 132 132 132 217 217 217 F F F 8 160-179 128 128 128 212 212 212 T F T <--- 9 180-199 130 130 130 215 215 215 T F T <--- 10 200-219 123 123 123 198 198 198 F F F 11 220-239 132 132 132 217 217 217 F F F 12 240-259 132 132 132 218 218 218 F F F 13 260-279 132 132 132 218 218 218 F F F 14 280-299 132 132 132 217 217 217 F F F 15 300-319 134 132 134 222 219 222 F F F <--- 16 320-339 128 128 128 213 213 213 F F F 17 340-359 128 128 128 213 213 213 T F T <--- 18 360-379 130 130 130 215 215 215 T F T <--- 19 380-399 134 133 134 221 220 221 F F F <--- 20 400-419 134 132 134 221 217 221 F F F <--- 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 128 128 128 213 213 213 T F T <--- 25 500-519 134 130 134 222 215 222 F F F <--- 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 131 134 221 216 221 F F F <--- Max gen: 134, Max flush: 222 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 127 127 127 209 209 209 F F F 1 020-039 134 134 134 222 222 222 F F F 2 040-059 128 128 128 213 213 213 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 128 128 128 212 212 212 F F F 5 100-119 135 135 135 223 223 223 F F F 6 120-139 135 135 135 223 223 223 F F F 7 140-159 132 132 132 217 217 217 F F F 8 160-179 135 135 135 223 223 223 F F F 9 180-199 135 135 135 223 223 223 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 132 132 132 217 217 217 F F F 12 240-259 132 132 132 218 218 218 F F F 13 260-279 132 132 132 218 218 218 F F F 14 280-299 132 132 132 217 217 217 F F F 15 300-319 134 134 134 222 222 222 F F F 16 320-339 128 128 128 213 213 213 F F F 17 340-359 135 135 135 223 223 223 F F F 18 360-379 135 135 135 223 223 223 F F F 19 380-399 134 134 134 221 221 221 F F F 20 400-419 134 134 134 221 221 221 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 135 135 135 223 223 223 F F F 25 500-519 134 134 134 222 222 222 F F F 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 135, Max flush: 223 Begin loop 68 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.282430903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.29062006Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.290900915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.290925053Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.290933552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.290944881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Crucible 5423f559-1e05-4440-b28d-9887f3f9643d has session id: dfb9b7d1-a382-428c-9eb9-45d253d03921","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291358725Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Upstairs opts: Upstairs UUID: 5423f559-1e05-4440-b28d-9887f3f9643d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291389693Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Crucible stats registered with UUID: 5423f559-1e05-4440-b28d-9887f3f9643d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291400832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:00.291410071Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291542179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291558388Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291568377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291729923Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291746362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"5423f559-1e05-4440-b28d-9887f3f9643d active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291781489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291790618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291800147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291809536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291818765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291828534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291838094Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"ds_connection connected from Ok(127.0.0.1:57597)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291915047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"ds_connection connected from Ok(127.0.0.1:49828)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291942065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"ds_connection connected from Ok(127.0.0.1:62829)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.291953644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.292627305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.292651423Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.29268031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.292923149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.292957476Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[0]R flush_numbers[0..12]: [209, 222, 213, 217, 212, 223, 223, 217, 223, 223, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293015361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[0]R generation[0..12]: [127, 134, 128, 132, 128, 135, 135, 132, 135, 135, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.29302649Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293037489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[1]R flush_numbers[0..12]: [209, 222, 213, 217, 212, 223, 223, 217, 223, 223, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293048008Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[1]R generation[0..12]: [127, 134, 128, 132, 128, 135, 135, 132, 135, 135, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293058357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293069186Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[2]R flush_numbers[0..12]: [209, 222, 213, 217, 212, 223, 223, 217, 223, 223, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293079565Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[2]R generation[0..12]: [127, 134, 128, 132, 128, 135, 135, 132, 135, 135, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293091964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293102803Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293112103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293121622Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Next flush: 224","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293130721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.29313989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"":"downstairs","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293149409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293157939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"0","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293167478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"1","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293177117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"client":"2","":"downstairs","session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293186816Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293195935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"5423f559-1e05-4440-b28d-9887f3f9643d is now active with session: dfb9b7d1-a382-428c-9eb9-45d253d03921","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293205774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293214194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"session_id":"dfb9b7d1-a382-428c-9eb9-45d253d03921"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293223143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Activated sub_volume 5423f559-1e05-4440-b28d-9887f3f9643d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293231022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293239951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.293551704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} 01/30 Write block 505 len 28672 data: 10 13 14 12 11 10 10 02/30 Read block 311 len 12288 03/30 Read block 40 len 40960 04/30 Read block 516 len 24576 05/30 Read block 525 len 36864 06/30 Read block 305 len 32768 07/30 Write block 43 len 40960 data: 5 10 9 10 10 14 13 13 12 11 08/30 Read block 498 len 4096 09/30 Write block 17 len 32768 data: 9 9 10 8 5 7 7 10 10/30 Write block 15 len 28672 data: 12 9 10 10 11 9 6 11/30 Read block 274 len 36864 12/30 Read block 106 len 8192 13/30 Read block 579 len 8192 14/30 Write block 98 len 32768 data: 7 8 9 10 10 12 13 12 15/30 Write block 389 len 28672 data: 19 17 17 23 17 19 17 16/30 Write block 325 len 8192 data: 12 13 17/30 Read block 180 len 32768 18/30 Read block 339 len 24576 19/30 Write block 306 len 8192 data: 13 13 20/30 Write block 44 len 16384 data: 11 10 11 11 21/30 Read block 401 len 36864 22/30 Read block 532 len 28672 23/30 Read block 495 len 24576 24/30 Flush 25/30 Write block 276 len 12288 data: 8 9 11 26/30 Read block 259 len 20480 27/30 Write block 175 len 40960 data: 7 10 10 8 8 8 6 6 9 10 28/30 Write block 25 len 24576 data: 10 10 9 10 7 7 29/30 Write block 345 len 28672 data: 13 11 9 8 10 13 11 30/30 Write block 276 len 28672 data: 9 10 12 12 12 11 10 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 6 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 30 3 36 Done 27 0 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:00.302339669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:00.302511754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1683,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 127 136 224 209 224 F F F <--- 1 020-039 136 134 136 224 222 224 T F T <--- 2 040-059 136 128 136 224 213 224 F F F <--- 3 060-079 132 132 132 217 217 217 F F F 4 080-099 136 128 136 224 212 224 F F F <--- 5 100-119 136 135 136 224 223 224 F F F <--- 6 120-139 135 135 135 223 223 223 F F F 7 140-159 132 132 132 217 217 217 F F F 8 160-179 135 135 135 223 223 223 T F T <--- 9 180-199 135 135 135 223 223 223 T F T <--- 10 200-219 123 123 123 198 198 198 F F F 11 220-239 132 132 132 217 217 217 F F F 12 240-259 132 132 132 218 218 218 F F F 13 260-279 132 132 132 218 218 218 T F T <--- 14 280-299 132 132 132 217 217 217 T F T <--- 15 300-319 136 134 136 224 222 224 F F F <--- 16 320-339 136 128 136 224 213 224 F F F <--- 17 340-359 135 135 135 223 223 223 T F T <--- 18 360-379 135 135 135 223 223 223 F F F 19 380-399 136 134 136 224 221 224 F F F <--- 20 400-419 134 134 134 221 221 221 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 135 135 135 223 223 223 F F F 25 500-519 136 134 136 224 222 224 F F F <--- 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 136, Max flush: 224 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 224 224 224 F F F 1 020-039 137 137 137 225 225 225 F F F 2 040-059 136 136 136 224 224 224 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 136 136 136 224 224 224 F F F 5 100-119 136 136 136 224 224 224 F F F 6 120-139 135 135 135 223 223 223 F F F 7 140-159 132 132 132 217 217 217 F F F 8 160-179 137 137 137 225 225 225 F F F 9 180-199 137 137 137 225 225 225 F F F 10 200-219 123 123 123 198 198 198 F F F 11 220-239 132 132 132 217 217 217 F F F 12 240-259 132 132 132 218 218 218 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 137 137 137 225 225 225 F F F 15 300-319 136 136 136 224 224 224 F F F 16 320-339 136 136 136 224 224 224 F F F 17 340-359 137 137 137 225 225 225 F F F 18 360-379 135 135 135 223 223 223 F F F 19 380-399 136 136 136 224 224 224 F F F 20 400-419 134 134 134 221 221 221 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F F F 24 480-499 135 135 135 223 223 223 F F F 25 500-519 136 136 136 224 224 224 F F F 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 137, Max flush: 225 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.560909219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.568056357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.568107382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575093444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575343632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.57536879Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575377239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575388618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Crucible 601e225c-ee93-464b-891e-e93d660c95af has session id: 700b8ea9-1428-4c26-b81c-f698c39d7515","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.57570461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Upstairs opts: Upstairs UUID: 601e225c-ee93-464b-891e-e93d660c95af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575740837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Crucible stats registered with UUID: 601e225c-ee93-464b-891e-e93d660c95af","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575751956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.575763005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575904943Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.57593503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.575946159Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576076998Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576087517Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"601e225c-ee93-464b-891e-e93d660c95af active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576119704Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576129133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576138603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576148202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576157501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.57616983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576179509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"ds_connection connected from Ok(127.0.0.1:40223)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576189838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"ds_connection connected from Ok(127.0.0.1:55493)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576243263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"ds_connection connected from Ok(127.0.0.1:51475)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576346105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576847161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.576870979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577006247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577209249Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577233717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[0]R flush_numbers[0..12]: [224, 225, 224, 217, 224, 224, 223, 217, 225, 225, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577301361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[0]R generation[0..12]: [136, 137, 136, 132, 136, 136, 135, 132, 137, 137, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.5773124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577323509Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[1]R flush_numbers[0..12]: [224, 225, 224, 217, 224, 224, 223, 217, 225, 225, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577333848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[1]R generation[0..12]: [136, 137, 136, 132, 136, 136, 135, 132, 137, 137, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577344118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577354747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[2]R flush_numbers[0..12]: [224, 225, 224, 217, 224, 224, 223, 217, 225, 225, 198, 217]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577365176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[2]R generation[0..12]: [136, 137, 136, 132, 136, 136, 135, 132, 137, 137, 123, 132]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577375475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577386154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577397093Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577406632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Next flush: 226","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577415751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577424881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"":"downstairs","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.57743437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577443109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"0","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577452838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"1","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577462647Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"client":"2","":"downstairs","session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577472546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577481576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"601e225c-ee93-464b-891e-e93d660c95af is now active with session: 700b8ea9-1428-4c26-b81c-f698c39d7515","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577491485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577500174Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577509143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Activated sub_volume 601e225c-ee93-464b-891e-e93d660c95af","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577516922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577524842Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.577814567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} 01/30 Write block 461 len 28672 data: 10 11 14 13 12 11 9 02/30 Write block 130 len 36864 data: 15 17 14 12 12 12 13 12 9 03/30 Write block 397 len 24576 data: 12 14 15 17 19 18 04/30 Read block 301 len 20480 05/30 Read block 111 len 32768 06/30 Write block 414 len 8192 data: 12 11 07/30 Write block 122 len 8192 data: 11 11 08/30 Write block 315 len 20480 data: 9 11 11 11 14 09/30 Write block 470 len 16384 data: 12 9 9 10 10/30 Write block 241 len 36864 data: 9 9 9 8 8 10 13 13 12 11/30 Read block 57 len 8192 12/30 Write block 204 len 16384 data: 6 8 8 8 13/30 Read block 278 len 36864 14/30 Write block 125 len 16384 data: 10 11 12 18 15/30 Write block 368 len 16384 data: 16 14 10 11 16/30 Write block 136 len 16384 data: 14 13 10 11 17/30 Read block 15 len 16384 18/30 Read block 350 len 32768 19/30 Read block 60 len 16384 20/30 Write block 85 len 32768 data: 10 12 10 14 14 11 8 9 21/30 Read block 41 len 24576 22/30 Write block 88 len 40960 data: 15 15 12 9 10 8 8 6 4 6 23/30 Write block 7 len 12288 data: 10 11 13 24/30 Write block 224 len 40960 data: 5 6 9 10 13 12 13 14 14 13 25/30 Read block 35 len 36864 26/30 Read block 363 len 32768 27/30 Read block 97 len 4096 28/30 Read block 512 len 36864 29/30 Read block 260 len 28672 30/30 Write block 367 len 32768 data: 13 17 15 11 12 11 11 13 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585674772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:04.585851886Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"task":"crutest"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585940889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585953717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585962667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585971736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585980765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.585994544Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586003583Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586012462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586021322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586030231Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58603922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586048289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586057118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586065758Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586074477Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586083106Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586091765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586100535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586110934Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586119723Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586128522Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586137281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586146081Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58615486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586163709Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586172818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586181938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586191677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586201016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586210255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586219334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586228314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586237333Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586246412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586255441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58626527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58627394Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586282699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586291428Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586300187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586308837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586317666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586326345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586334974Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586343623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586352253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586360992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586369761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58637845Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58638707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586395749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586404748Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586414567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586423547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586432556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586441495Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586450424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586459363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586468363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586477212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586486201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58649519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58650395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586512679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586521438Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586530227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586538946Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586547686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586556365Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586566144Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586574823Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586583553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586592292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586601011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58660979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58661848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586627049Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586635988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586644877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586653796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586662736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586671705Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586680564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586689473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586698383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586707422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586717821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58672681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586735609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586744329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586753028Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586761757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586770466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586779016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586787555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586796304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586804983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586813663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586822332Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586831081Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58683979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.5868484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586857289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586867128Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586876027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586884956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586893896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586902645Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586911584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586920463Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586929422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586938282Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586947291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.58695626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586965059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586973709Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586982308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.586991077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587000116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587009056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587017775Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587027694Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587036393Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587045142Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587053982Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:04.587062671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1691,"session_id":"700b8ea9-1428-4c26-b81c-f698c39d7515"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 224 224 224 F T T <--- 1 020-039 137 137 137 225 225 225 F F F 2 040-059 136 136 136 224 224 224 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 136 136 136 224 224 224 F T T <--- 5 100-119 136 136 136 224 224 224 F F F 6 120-139 135 135 135 223 223 223 F T T <--- 7 140-159 132 132 132 217 217 217 F F F 8 160-179 137 137 137 225 225 225 F F F 9 180-199 137 137 137 225 225 225 F F F 10 200-219 123 123 123 198 198 198 F T T <--- 11 220-239 132 132 132 217 217 217 F T T <--- 12 240-259 132 132 132 218 218 218 F T T <--- 13 260-279 137 137 137 225 225 225 F F F 14 280-299 137 137 137 225 225 225 F F F 15 300-319 136 136 136 224 224 224 F T T <--- 16 320-339 136 136 136 224 224 224 F F F 17 340-359 137 137 137 225 225 225 F F F 18 360-379 135 135 135 223 223 223 F T T <--- 19 380-399 136 136 136 224 224 224 F T T <--- 20 400-419 134 134 134 221 221 221 F T T <--- 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 128 128 128 211 211 211 F T T <--- 24 480-499 135 135 135 223 223 223 F F F 25 500-519 136 136 136 224 224 224 F F F 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 137, Max flush: 225 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 137 137 137 225 225 225 F F F 2 040-059 136 136 136 224 224 224 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 138 138 138 226 226 226 F F F 5 100-119 136 136 136 224 224 224 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 132 132 132 217 217 217 F F F 8 160-179 137 137 137 225 225 225 F F F 9 180-199 137 137 137 225 225 225 F F F 10 200-219 138 138 138 226 226 226 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 137 137 137 225 225 225 F F F 15 300-319 138 138 138 226 226 226 F F F 16 320-339 136 136 136 224 224 224 F F F 17 340-359 137 137 137 225 225 225 F F F 18 360-379 138 138 138 226 226 226 F F F 19 380-399 138 138 138 226 226 226 F F F 20 400-419 138 138 138 226 226 226 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 131 131 216 216 216 F F F 23 460-479 138 138 138 226 226 226 F F F 24 480-499 135 135 135 223 223 223 F F F 25 500-519 136 136 136 224 224 224 F F F 26 520-539 133 133 133 220 220 220 F F F 27 540-559 132 132 132 217 217 217 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 138, Max flush: 226 Begin loop 70 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.843897312Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:08.851082256Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.851137801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858012313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.85826663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858293208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858301677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858313056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Crucible afab602b-3fc9-48f8-a3a0-236b2674bf22 has session id: 3de09c74-2416-4c22-a2b3-853f0493098c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858647677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Upstairs opts: Upstairs UUID: afab602b-3fc9-48f8-a3a0-236b2674bf22, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858678035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Crucible stats registered with UUID: afab602b-3fc9-48f8-a3a0-236b2674bf22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858688624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:08.858697573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858812753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858829171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.858839121Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859002226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859014625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"afab602b-3fc9-48f8-a3a0-236b2674bf22 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859030944Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859039613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859049192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859058571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859068101Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859107757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859132905Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"ds_connection connected from Ok(127.0.0.1:52682)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859143904Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"ds_connection connected from Ok(127.0.0.1:50104)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859156303Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"ds_connection connected from Ok(127.0.0.1:47963)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859166412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.85976501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859802727Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.859836424Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860100351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860123719Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[0]R flush_numbers[0..12]: [226, 225, 224, 217, 226, 224, 226, 217, 225, 225, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860204532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[0]R generation[0..12]: [138, 137, 136, 132, 138, 136, 138, 132, 137, 137, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860216191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.86022692Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[1]R flush_numbers[0..12]: [226, 225, 224, 217, 226, 224, 226, 217, 225, 225, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860237559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[1]R generation[0..12]: [138, 137, 136, 132, 138, 136, 138, 132, 137, 137, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860248058Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860258977Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[2]R flush_numbers[0..12]: [226, 225, 224, 217, 226, 224, 226, 217, 225, 225, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860269406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[2]R generation[0..12]: [138, 137, 136, 132, 138, 136, 138, 132, 137, 137, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860282265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860292924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Max found gen is 139","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860302173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Generation requested: 140 >= found:139","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860311682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Next flush: 227","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860320861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860329891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"":"downstairs","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.86033945Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860347989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"0","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860357768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"1","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860367527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"client":"2","":"downstairs","session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860377246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860386276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"afab602b-3fc9-48f8-a3a0-236b2674bf22 is now active with session: 3de09c74-2416-4c22-a2b3-853f0493098c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860396165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860404644Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"session_id":"3de09c74-2416-4c22-a2b3-853f0493098c"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860413613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Activated sub_volume afab602b-3fc9-48f8-a3a0-236b2674bf22","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860421383Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860430932Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.860733305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} 01/30 Write block 114 len 12288 data: 7 8 9 02/30 Read block 252 len 36864 03/30 Write block 405 len 16384 data: 16 16 15 14 04/30 Read block 521 len 24576 05/30 Write block 353 len 28672 data: 8 12 10 8 8 9 9 06/30 Write block 95 len 36864 data: 7 5 7 8 9 10 11 11 13 07/30 Flush 08/30 Write block 513 len 32768 data: 16 15 13 14 14 13 14 14 09/30 Read block 106 len 20480 10/30 Read block 183 len 32768 11/30 Write block 448 len 24576 data: 14 13 14 13 9 10 12/30 Flush 13/30 Write block 538 len 12288 data: 10 8 9 14/30 Write block 26 len 12288 data: 11 10 11 15/30 Read block 430 len 28672 16/30 Write block 478 len 16384 data: 14 12 10 12 17/30 Read block 305 len 32768 18/30 Write block 207 len 4096 data: 9 19/30 Read block 516 len 24576 20/30 Flush 21/30 Flush 22/30 Write block 453 len 8192 data: 11 10 23/30 Write block 86 len 40960 data: 13 11 16 16 13 10 11 9 9 8 24/30 Read block 203 len 32768 25/30 Write block 153 len 36864 data: 15 15 15 13 14 13 11 9 7 26/30 Write block 358 len 16384 data: 10 10 8 13 27/30 Write block 319 len 28672 data: 15 14 12 14 14 13 13 28/30 Read block 232 len 40960 29/30 Write block 394 len 40960 data: 20 18 13 13 15 16 18 20 19 15 30/30 Write block 351 len 4096 data: 12 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:08.869783357Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:08.86997532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1699,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 137 140 140 225 229 229 F F F <--- 2 040-059 136 136 136 224 224 224 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 138 140 140 226 227 227 F T T <--- 5 100-119 136 140 140 224 227 227 F F F <--- 6 120-139 138 138 138 226 226 226 F F F 7 140-159 132 132 132 217 217 217 F T T <--- 8 160-179 137 137 137 225 225 225 F T T <--- 9 180-199 137 137 137 225 225 225 F F F 10 200-219 138 140 140 226 229 229 F F F <--- 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 137 137 137 225 225 225 F F F 15 300-319 138 138 138 226 226 226 F T T <--- 16 320-339 136 136 136 224 224 224 F T T <--- 17 340-359 137 140 140 225 227 227 F T T <--- 18 360-379 138 138 138 226 226 226 F T T <--- 19 380-399 138 138 138 226 226 226 F T T <--- 20 400-419 138 140 140 226 227 227 F T T <--- 21 420-439 131 131 131 216 216 216 F F F 22 440-459 131 140 140 216 228 228 F T T <--- 23 460-479 138 140 140 226 229 229 F F F <--- 24 480-499 135 140 140 223 229 229 F F F <--- 25 500-519 136 140 140 224 228 228 F F F <--- 26 520-539 133 140 140 220 229 229 F F F <--- 27 540-559 132 140 140 217 229 229 F F F <--- 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 140, Max flush: 229 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 140 140 140 229 229 229 F F F 2 040-059 136 136 136 224 224 224 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 141 141 141 230 230 230 F F F 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 141 141 141 230 230 230 F F F 8 160-179 141 141 141 230 230 230 F F F 9 180-199 137 137 137 225 225 225 F F F 10 200-219 140 140 140 229 229 229 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 137 137 137 225 225 225 F F F 15 300-319 141 141 141 230 230 230 F F F 16 320-339 141 141 141 230 230 230 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 141 141 141 230 230 230 F F F 19 380-399 141 141 141 230 230 230 F F F 20 400-419 141 141 141 230 230 230 F F F 21 420-439 131 131 131 216 216 216 F F F 22 440-459 141 141 141 230 230 230 F F F 23 460-479 140 140 140 229 229 229 F F F 24 480-499 140 140 140 229 229 229 F F F 25 500-519 140 140 140 228 228 228 F F F 26 520-539 140 140 140 229 229 229 F F F 27 540-559 140 140 140 229 229 229 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 141, Max flush: 230 Begin loop 71 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.14438261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.152586616Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.152866011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.152893979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.152902488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.152913867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Crucible 2635fe66-2a27-4235-8305-d4deddfa66dc has session id: 0bcf3b90-914b-4ca6-85c6-4ce73c900348","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153239799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Upstairs opts: Upstairs UUID: 2635fe66-2a27-4235-8305-d4deddfa66dc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153315762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Crucible stats registered with UUID: 2635fe66-2a27-4235-8305-d4deddfa66dc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153325701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:13.15334238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153473178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153489067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153501146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153690389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153706838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"2635fe66-2a27-4235-8305-d4deddfa66dc active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153739295Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153748444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153757964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153767453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153776852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153786591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.15379621Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"ds_connection connected from Ok(127.0.0.1:35980)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153808709Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"ds_connection connected from Ok(127.0.0.1:59721)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153851385Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"ds_connection connected from Ok(127.0.0.1:36691)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.153861994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154498699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154519027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154553744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154823541Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154847609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[0]R flush_numbers[0..12]: [226, 229, 224, 217, 230, 227, 226, 230, 230, 225, 229, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154895034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[0]R generation[0..12]: [138, 140, 136, 132, 141, 140, 138, 141, 141, 137, 140, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154906773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154918002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[1]R flush_numbers[0..12]: [226, 229, 224, 217, 230, 227, 226, 230, 230, 225, 229, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154928482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[1]R generation[0..12]: [138, 140, 136, 132, 141, 140, 138, 141, 141, 137, 140, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154938801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.15494954Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[2]R flush_numbers[0..12]: [226, 229, 224, 217, 230, 227, 226, 230, 230, 225, 229, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154960219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[2]R generation[0..12]: [138, 140, 136, 132, 141, 140, 138, 141, 141, 137, 140, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154970678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154981487Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.154990786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155000455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Next flush: 231","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155012414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155021653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"":"downstairs","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155031083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155039702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"0","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155049291Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"1","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.15505861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"client":"2","":"downstairs","session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155068279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155077239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"2635fe66-2a27-4235-8305-d4deddfa66dc is now active with session: 0bcf3b90-914b-4ca6-85c6-4ce73c900348","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155086878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155095567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"session_id":"0bcf3b90-914b-4ca6-85c6-4ce73c900348"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155104446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Activated sub_volume 2635fe66-2a27-4235-8305-d4deddfa66dc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155112306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155120225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.155447436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} 01/30 Flush 02/30 Write block 295 len 40960 data: 14 15 14 13 16 12 11 14 16 16 03/30 Read block 387 len 8192 04/30 Write block 437 len 28672 data: 10 12 11 13 13 15 13 05/30 Flush 06/30 Write block 436 len 12288 data: 9 11 13 07/30 Write block 464 len 20480 data: 14 13 12 10 9 08/30 Write block 217 len 8192 data: 11 12 09/30 Read block 392 len 24576 10/30 Write block 313 len 32768 data: 9 10 10 12 12 12 16 15 11/30 Write block 360 len 16384 data: 9 14 13 14 12/30 Write block 91 len 28672 data: 11 12 10 10 9 6 8 13/30 Write block 427 len 8192 data: 8 12 14/30 Read block 298 len 4096 15/30 Read block 243 len 36864 16/30 Write block 170 len 12288 data: 7 8 8 17/30 Read block 271 len 4096 18/30 Read block 32 len 12288 19/30 Write block 369 len 4096 data: 16 20/30 Flush 21/30 Flush 22/30 Flush 23/30 Write block 54 len 20480 data: 13 11 9 8 8 24/30 Read block 186 len 40960 25/30 Read block 466 len 20480 26/30 Write block 85 len 36864 data: 11 14 12 17 17 14 12 13 11 27/30 Write block 46 len 40960 data: 12 12 15 14 14 13 12 11 14 12 28/30 Read block 135 len 8192 29/30 Read block 147 len 28672 30/30 Read block 294 len 16384 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Sent Done false 1001 Acked Write 10 Done Sent Done false 1002 Acked Read 2 Done Sent Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:13.163704107Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:13.163875452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1707,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 140 140 140 229 229 229 F F F 2 040-059 136 136 136 224 224 224 T F T <--- 3 060-079 132 132 132 217 217 217 F F F 4 080-099 142 141 142 233 230 233 T F T <--- 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 141 141 141 230 230 230 F F F 8 160-179 142 141 142 233 230 233 F F F <--- 9 180-199 137 137 137 225 225 225 F F F 10 200-219 142 140 142 233 229 233 F F F <--- 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 142 137 142 232 225 232 F F F <--- 15 300-319 142 141 142 233 230 233 F F F <--- 16 320-339 142 141 142 233 230 233 F F F <--- 17 340-359 141 141 141 230 230 230 F F F 18 360-379 142 141 142 233 230 233 F F F <--- 19 380-399 141 141 141 230 230 230 F F F 20 400-419 141 141 141 230 230 230 F F F 21 420-439 142 131 142 233 216 233 F F F <--- 22 440-459 142 141 142 232 230 232 F F F <--- 23 460-479 142 140 142 233 229 233 F F F <--- 24 480-499 140 140 140 229 229 229 F F F 25 500-519 140 140 140 228 228 228 F F F 26 520-539 140 140 140 229 229 229 F F F 27 540-559 140 140 140 229 229 229 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 142, Max flush: 233 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 140 140 140 229 229 229 F F F 2 040-059 143 143 143 234 234 234 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 143 143 143 234 234 234 F F F 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 141 141 141 230 230 230 F F F 8 160-179 142 142 142 233 233 233 F F F 9 180-199 137 137 137 225 225 225 F F F 10 200-219 142 142 142 233 233 233 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 137 137 137 225 225 225 F F F 14 280-299 142 142 142 232 232 232 F F F 15 300-319 142 142 142 233 233 233 F F F 16 320-339 142 142 142 233 233 233 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 142 142 142 233 233 233 F F F 19 380-399 141 141 141 230 230 230 F F F 20 400-419 141 141 141 230 230 230 F F F 21 420-439 142 142 142 233 233 233 F F F 22 440-459 142 142 142 232 232 232 F F F 23 460-479 142 142 142 233 233 233 F F F 24 480-499 140 140 140 229 229 229 F F F 25 500-519 140 140 140 228 228 228 F F F 26 520-539 140 140 140 229 229 229 F F F 27 540-559 140 140 140 229 229 229 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 143, Max flush: 234 Begin loop 72 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.410734233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.41891634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419204685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419230613Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419241552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419253041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Crucible 7c266261-24f0-43a4-b9fb-9e5ede4f454f has session id: f60abf7f-d9dd-444a-8683-7859e47819f0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419537016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Upstairs opts: Upstairs UUID: 7c266261-24f0-43a4-b9fb-9e5ede4f454f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419601611Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Crucible stats registered with UUID: 7c266261-24f0-43a4-b9fb-9e5ede4f454f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.41961189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:17.419620889Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419772306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419803573Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419814212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419934362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.41995781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"7c266261-24f0-43a4-b9fb-9e5ede4f454f active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419977378Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.419986427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420006855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420018344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420027773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420037763Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.42007023Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:50847)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420092138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:34559)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420103247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:46934)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420113686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420741341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420804756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.420879639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421073792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421090511Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[0]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 230, 233, 225, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421197601Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[0]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 141, 142, 137, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421209Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.42122024Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[1]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 230, 233, 225, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421233348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[1]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 141, 142, 137, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421243897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421254747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[2]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 230, 233, 225, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421265156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[2]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 141, 142, 137, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421275485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421286204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421295273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421305022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421314091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421323221Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"":"downstairs","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.42133274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421341439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"0","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421350868Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"1","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421360267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"client":"2","":"downstairs","session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421371236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421380236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"7c266261-24f0-43a4-b9fb-9e5ede4f454f is now active with session: f60abf7f-d9dd-444a-8683-7859e47819f0","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421389935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421398474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"session_id":"f60abf7f-d9dd-444a-8683-7859e47819f0"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421407313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Activated sub_volume 7c266261-24f0-43a4-b9fb-9e5ede4f454f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421415133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421423082Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.421774711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} 01/30 Write block 299 len 36864 data: 17 13 12 15 17 17 15 14 14 02/30 Write block 148 len 16384 data: 8 11 11 12 03/30 Read block 541 len 36864 04/30 Read block 436 len 12288 05/30 Read block 186 len 4096 06/30 Write block 401 len 20480 data: 21 20 16 17 17 07/30 Flush 08/30 Write block 434 len 36864 data: 10 10 10 12 14 12 14 14 16 09/30 Write block 374 len 16384 data: 14 14 15 16 10/30 Read block 474 len 12288 11/30 Read block 189 len 4096 12/30 Write block 382 len 16384 data: 13 16 13 13 13/30 Write block 554 len 20480 data: 8 12 10 9 10 14/30 Write block 504 len 12288 data: 10 11 14 15/30 Write block 520 len 36864 data: 15 14 14 12 14 11 10 11 10 16/30 Read block 326 len 20480 17/30 Read block 19 len 36864 18/30 Read block 222 len 24576 19/30 Read block 0 len 40960 20/30 Flush 21/30 Write block 265 len 40960 data: 15 16 13 13 13 15 14 14 14 13 22/30 Read block 127 len 12288 23/30 Flush 24/30 Write block 492 len 8192 data: 7 8 25/30 Write block 460 len 20480 data: 11 11 12 15 15 26/30 Write block 190 len 20480 data: 9 10 10 6 6 27/30 Write block 508 len 40960 data: 13 12 11 11 14 17 16 14 15 15 28/30 Write block 310 len 28672 data: 13 10 10 10 11 11 13 29/30 Read block 207 len 32768 30/30 Read block 200 len 28672 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 1 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 26 27 Done 30 29 4 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:17.430453345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:17.430635409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1715,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 140 140 140 229 229 229 F F F 2 040-059 143 143 143 234 234 234 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 143 143 143 234 234 234 F F F 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 144 144 141 235 235 230 F F T <--- 8 160-179 142 142 142 233 233 233 F F F 9 180-199 137 137 137 225 225 225 T T F <--- 10 200-219 142 142 142 233 233 233 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 144 144 137 237 237 225 F F F <--- 14 280-299 144 144 142 235 235 232 F F T <--- 15 300-319 144 144 142 235 235 233 T T T <--- 16 320-339 142 142 142 233 233 233 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 142 236 236 233 F F F <--- 19 380-399 144 144 141 236 236 230 F F F <--- 20 400-419 144 144 141 235 235 230 F F F <--- 21 420-439 144 144 142 236 236 233 F F F <--- 22 440-459 144 144 142 236 236 232 F F F <--- 23 460-479 142 142 142 233 233 233 T T F <--- 24 480-499 140 140 140 229 229 229 T T F <--- 25 500-519 144 144 140 236 236 228 T T F <--- 26 520-539 144 144 140 236 236 229 F F F <--- 27 540-559 144 144 140 236 236 229 F F F <--- 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 144, Max flush: 237 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 F F F 1 020-039 140 140 140 229 229 229 F F F 2 040-059 143 143 143 234 234 234 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 143 143 143 234 234 234 F F F 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 144 144 144 235 235 235 F F F 8 160-179 142 142 142 233 233 233 F F F 9 180-199 145 145 145 238 238 238 F F F 10 200-219 142 142 142 233 233 233 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 144 144 144 237 237 237 F F F 14 280-299 144 144 144 235 235 235 F F F 15 300-319 145 145 145 238 238 238 F F F 16 320-339 142 142 142 233 233 233 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 144 236 236 236 F F F 19 380-399 144 144 144 236 236 236 F F F 20 400-419 144 144 144 235 235 235 F F F 21 420-439 144 144 144 236 236 236 F F F 22 440-459 144 144 144 236 236 236 F F F 23 460-479 145 145 145 238 238 238 F F F 24 480-499 145 145 145 238 238 238 F F F 25 500-519 145 145 145 238 238 238 F F F 26 520-539 144 144 144 236 236 236 F F F 27 540-559 144 144 144 236 236 236 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 F F F Max gen: 145, Max flush: 238 Begin loop 73 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.69276147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.700963376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.70125069Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701276148Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701284668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701296346Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Crucible 4684a681-c804-4429-a3f2-0f4d16d26d25 has session id: d3504b07-45c0-4362-8b7f-49720ab4e2d8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701650176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Upstairs opts: Upstairs UUID: 4684a681-c804-4429-a3f2-0f4d16d26d25, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701677323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Crucible stats registered with UUID: 4684a681-c804-4429-a3f2-0f4d16d26d25","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701687092Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:21.701707651Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701857238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701886285Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.701896964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702042701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702066699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"4684a681-c804-4429-a3f2-0f4d16d26d25 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702076049Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702084798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702094347Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702103706Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702113165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702122994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702132534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:48756)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702215736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:57900)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702241824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:41718)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702253133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702879449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702897097Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.702924855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703233568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703243237Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[0]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 235, 233, 238, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703278154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[0]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 144, 142, 145, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703289083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703299892Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[1]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 235, 233, 238, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703310401Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[1]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 144, 142, 145, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.70332085Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703331929Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[2]R flush_numbers[0..12]: [226, 229, 234, 217, 234, 227, 226, 235, 233, 238, 233, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703342458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[2]R generation[0..12]: [138, 140, 143, 132, 143, 140, 138, 144, 142, 145, 142, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703354937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703365736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703374965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703384535Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Next flush: 239","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703393604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703402793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"":"downstairs","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703412262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703420911Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"0","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703430201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"1","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.70343961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"client":"2","":"downstairs","session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703449339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703458218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"4684a681-c804-4429-a3f2-0f4d16d26d25 is now active with session: d3504b07-45c0-4362-8b7f-49720ab4e2d8","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703467827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703476367Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"session_id":"d3504b07-45c0-4362-8b7f-49720ab4e2d8"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703485296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Activated sub_volume 4684a681-c804-4429-a3f2-0f4d16d26d25","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703493215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703502294Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.703818397Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} 01/30 Flush 02/30 Write block 387 len 32768 data: 20 19 20 18 18 24 18 21 03/30 Read block 549 len 20480 04/30 Read block 172 len 32768 05/30 Read block 78 len 12288 06/30 Read block 443 len 8192 07/30 Read block 376 len 36864 08/30 Read block 201 len 20480 09/30 Read block 29 len 32768 10/30 Write block 0 len 36864 data: 3 4 7 8 7 8 9 11 12 11/30 Read block 49 len 24576 12/30 Write block 159 len 16384 data: 12 10 8 9 13/30 Write block 193 len 40960 data: 7 7 8 7 7 6 4 3 4 5 14/30 Write block 416 len 4096 data: 16 15/30 Write block 584 len 20480 data: 11 12 15 13 14 16/30 Write block 188 len 4096 data: 5 17/30 Write block 95 len 12288 data: 10 7 9 18/30 Read block 522 len 16384 19/30 Write block 417 len 16384 data: 14 16 14 15 20/30 Write block 520 len 8192 data: 16 15 21/30 Write block 160 len 4096 data: 11 22/30 Write block 397 len 36864 data: 14 16 17 19 22 21 17 18 18 23/30 Read block 216 len 4096 24/30 Write block 17 len 20480 data: 11 11 12 10 7 25/30 Write block 281 len 40960 data: 12 11 9 10 10 10 10 10 9 9 26/30 Write block 476 len 8192 data: 11 11 27/30 Read block 145 len 24576 28/30 Write block 407 len 32768 data: 16 15 12 12 9 9 9 13 29/30 Write block 338 len 4096 data: 16 30/30 Write block 503 len 24576 data: 8 11 12 15 15 14 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 29 35 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:21.711085204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:21.711282756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1723,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 226 226 226 T T F <--- 1 020-039 140 140 140 229 229 229 T T F <--- 2 040-059 143 143 143 234 234 234 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 143 143 143 234 234 234 T T F <--- 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 144 144 144 235 235 235 T T F <--- 8 160-179 142 142 142 233 233 233 T T F <--- 9 180-199 145 145 145 238 238 238 T T F <--- 10 200-219 142 142 142 233 233 233 T T F <--- 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 144 144 144 237 237 237 F F F 14 280-299 144 144 144 235 235 235 T T F <--- 15 300-319 145 145 145 238 238 238 F F F 16 320-339 142 142 142 233 233 233 T T F <--- 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 144 236 236 236 F F F 19 380-399 144 144 144 236 236 236 T T F <--- 20 400-419 144 144 144 235 235 235 T T F <--- 21 420-439 144 144 144 236 236 236 T T F <--- 22 440-459 144 144 144 236 236 236 F F F 23 460-479 145 145 145 238 238 238 T T F <--- 24 480-499 145 145 145 238 238 238 F F F 25 500-519 145 145 145 238 238 238 T T F <--- 26 520-539 144 144 144 236 236 236 T T F <--- 27 540-559 144 144 144 236 236 236 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 134 134 134 221 221 221 T T F <--- Max gen: 145, Max flush: 238 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 239 239 239 F F F 1 020-039 146 146 146 239 239 239 F F F 2 040-059 143 143 143 234 234 234 F F F 3 060-079 132 132 132 217 217 217 F F F 4 080-099 146 146 146 239 239 239 F F F 5 100-119 140 140 140 227 227 227 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 146 146 146 239 239 239 F F F 9 180-199 146 146 146 239 239 239 F F F 10 200-219 146 146 146 239 239 239 F F F 11 220-239 138 138 138 226 226 226 F F F 12 240-259 138 138 138 226 226 226 F F F 13 260-279 144 144 144 237 237 237 F F F 14 280-299 146 146 146 239 239 239 F F F 15 300-319 145 145 145 238 238 238 F F F 16 320-339 146 146 146 239 239 239 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 144 236 236 236 F F F 19 380-399 146 146 146 239 239 239 F F F 20 400-419 146 146 146 239 239 239 F F F 21 420-439 146 146 146 239 239 239 F F F 22 440-459 144 144 144 236 236 236 F F F 23 460-479 146 146 146 239 239 239 F F F 24 480-499 145 145 145 238 238 238 F F F 25 500-519 146 146 146 239 239 239 F F F 26 520-539 146 146 146 239 239 239 F F F 27 540-559 144 144 144 236 236 236 F F F 28 560-579 129 129 129 214 214 214 F F F 29 580-599 146 146 146 239 239 239 F F F Max gen: 146, Max flush: 239 Begin loop 74 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.9776239Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.985896099Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986243989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986269517Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986278116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986289455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Crucible 9f083c17-1a78-4c31-9dc2-fa3ceb8098d5 has session id: 9b768219-5413-4b28-90a2-6205c2792c1d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986652653Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Upstairs opts: Upstairs UUID: 9f083c17-1a78-4c31-9dc2-fa3ceb8098d5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986680481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Crucible stats registered with UUID: 9f083c17-1a78-4c31-9dc2-fa3ceb8098d5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.98669111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:25.986700369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986840297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986861615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.986871484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.98703528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987052848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"9f083c17-1a78-4c31-9dc2-fa3ceb8098d5 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987069217Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987077856Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987087195Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987096715Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987106014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987115873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987125482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:52239)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987238242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47161)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987386559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46230)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987411897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987820711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987851379Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.987988797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988129735Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988153972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[0]R flush_numbers[0..12]: [239, 239, 234, 217, 239, 227, 226, 239, 239, 239, 239, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988247574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[0]R generation[0..12]: [146, 146, 143, 132, 146, 140, 138, 146, 146, 146, 146, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988258523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988269262Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[1]R flush_numbers[0..12]: [239, 239, 234, 217, 239, 227, 226, 239, 239, 239, 239, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988279711Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[1]R generation[0..12]: [146, 146, 143, 132, 146, 140, 138, 146, 146, 146, 146, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988289981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.98830074Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[2]R flush_numbers[0..12]: [239, 239, 234, 217, 239, 227, 226, 239, 239, 239, 239, 226]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988311389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[2]R generation[0..12]: [146, 146, 143, 132, 146, 140, 138, 146, 146, 146, 146, 138]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988321688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988332427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Max found gen is 147","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988341656Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Generation requested: 148 >= found:147","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988351165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Next flush: 240","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988361994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988371114Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"":"downstairs","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988380453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988389062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"0","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988398471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"1","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.98840788Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"client":"2","":"downstairs","session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988417429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988426279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"9f083c17-1a78-4c31-9dc2-fa3ceb8098d5 is now active with session: 9b768219-5413-4b28-90a2-6205c2792c1d","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988436078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988444767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"session_id":"9b768219-5413-4b28-90a2-6205c2792c1d"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988453586Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Activated sub_volume 9f083c17-1a78-4c31-9dc2-fa3ceb8098d5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988461546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988469525Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.988853052Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} 01/30 Write block 50 len 32768 data: 15 14 13 12 15 13 10 9 02/30 Read block 187 len 8192 03/30 Read block 40 len 36864 04/30 Read block 65 len 36864 05/30 Write block 301 len 8192 data: 13 16 06/30 Read block 457 len 8192 07/30 Read block 32 len 32768 08/30 Write block 234 len 24576 data: 15 18 15 15 15 13 09/30 Write block 99 len 4096 data: 10 10/30 Write block 570 len 20480 data: 7 9 10 9 9 11/30 Write block 427 len 16384 data: 9 13 12 13 12/30 Write block 264 len 8192 data: 13 16 13/30 Write block 272 len 28672 data: 15 15 14 10 10 11 13 14/30 Write block 271 len 36864 data: 15 16 16 15 11 11 12 14 13 15/30 Read block 97 len 28672 16/30 Write block 585 len 24576 data: 13 16 14 15 15 16 17/30 Flush 18/30 Write block 516 len 36864 data: 16 16 14 15 17 16 15 13 15 19/30 Write block 84 len 36864 data: 10 12 15 13 18 18 15 13 14 20/30 Read block 292 len 4096 21/30 Write block 68 len 16384 data: 10 9 9 9 22/30 Write block 66 len 40960 data: 10 11 11 10 10 10 8 7 7 8 23/30 Read block 353 len 32768 24/30 Read block 406 len 32768 25/30 Read block 445 len 32768 26/30 Write block 251 len 28672 data: 8 6 8 10 9 9 9 27/30 Write block 208 len 4096 data: 8 28/30 Read block 293 len 28672 29/30 Write block 105 len 16384 data: 13 10 10 8 30/30 Write block 565 len 32768 data: 8 8 6 6 10 8 10 11 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 2 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:25.997126151Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:25.997300486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1731,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 239 239 239 F F F 1 020-039 146 146 146 239 239 239 F F F 2 040-059 148 148 143 240 240 234 F F T <--- 3 060-079 132 132 132 217 217 217 T T F <--- 4 080-099 148 148 146 240 240 239 T T F <--- 5 100-119 140 140 140 227 227 227 T T F <--- 6 120-139 138 138 138 226 226 226 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 146 146 146 239 239 239 F F F 9 180-199 146 146 146 239 239 239 F F F 10 200-219 146 146 146 239 239 239 T T F <--- 11 220-239 148 148 138 240 240 226 F F F <--- 12 240-259 138 138 138 226 226 226 T T F <--- 13 260-279 148 148 144 240 240 237 F F F <--- 14 280-299 146 146 146 239 239 239 F F F 15 300-319 148 148 145 240 240 238 F F F <--- 16 320-339 146 146 146 239 239 239 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 144 236 236 236 F F F 19 380-399 146 146 146 239 239 239 F F F 20 400-419 146 146 146 239 239 239 F F F 21 420-439 148 148 146 240 240 239 F F F <--- 22 440-459 144 144 144 236 236 236 F F F 23 460-479 146 146 146 239 239 239 F F F 24 480-499 145 145 145 238 238 238 F F F 25 500-519 146 146 146 239 239 239 T T F <--- 26 520-539 146 146 146 239 239 239 T T F <--- 27 540-559 144 144 144 236 236 236 F F F 28 560-579 148 148 129 240 240 214 T T F <--- 29 580-599 148 148 146 240 240 239 F F F <--- Max gen: 148, Max flush: 240 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 239 239 239 F F F 1 020-039 146 146 146 239 239 239 F F F 2 040-059 148 148 148 240 240 240 F F F 3 060-079 149 149 149 241 241 241 F F F 4 080-099 149 149 149 241 241 241 F F F 5 100-119 149 149 149 241 241 241 F F F 6 120-139 138 138 138 226 226 226 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 146 146 146 239 239 239 F F F 9 180-199 146 146 146 239 239 239 F F F 10 200-219 149 149 149 241 241 241 F F F 11 220-239 148 148 148 240 240 240 F F F 12 240-259 149 149 149 241 241 241 F F F 13 260-279 148 148 148 240 240 240 F F F 14 280-299 146 146 146 239 239 239 F F F 15 300-319 148 148 148 240 240 240 F F F 16 320-339 146 146 146 239 239 239 F F F 17 340-359 141 141 141 230 230 230 F F F 18 360-379 144 144 144 236 236 236 F F F 19 380-399 146 146 146 239 239 239 F F F 20 400-419 146 146 146 239 239 239 F F F 21 420-439 148 148 148 240 240 240 F F F 22 440-459 144 144 144 236 236 236 F F F 23 460-479 146 146 146 239 239 239 F F F 24 480-499 145 145 145 238 238 238 F F F 25 500-519 149 149 149 241 241 241 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 144 144 144 236 236 236 F F F 28 560-579 149 149 149 241 241 241 F F F 29 580-599 148 148 148 240 240 240 F F F Max gen: 149, Max flush: 241 Begin loop 75 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.257082441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:30.264246047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.264294142Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271293453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27154947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271577038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271585637Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271597176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Crucible 120119d8-c727-4ae0-8115-6cf5c897a6ca has session id: 71ae7220-101b-4ce8-859e-5c19c5fad2e9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271931967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Upstairs opts: Upstairs UUID: 120119d8-c727-4ae0-8115-6cf5c897a6ca, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271960164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Crucible stats registered with UUID: 120119d8-c727-4ae0-8115-6cf5c897a6ca","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.271970964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:30.271979983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272093113Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272109631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272119571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272285076Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272309014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"120119d8-c727-4ae0-8115-6cf5c897a6ca active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272324223Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272332912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272342621Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27235216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272361669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272371559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272381298Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:63786)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272433963Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:39620)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.272458981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:39520)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27246992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273105585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273132782Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273143631Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273420157Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273459914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[0]R flush_numbers[0..12]: [239, 239, 240, 241, 241, 241, 226, 239, 239, 239, 241, 240]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273522718Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[0]R generation[0..12]: [146, 146, 148, 149, 149, 149, 138, 146, 146, 146, 149, 148]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273533837Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273544636Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[1]R flush_numbers[0..12]: [239, 239, 240, 241, 241, 241, 226, 239, 239, 239, 241, 240]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273555115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[1]R generation[0..12]: [146, 146, 148, 149, 149, 149, 138, 146, 146, 146, 149, 148]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273565365Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273576084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[2]R flush_numbers[0..12]: [239, 239, 240, 241, 241, 241, 226, 239, 239, 239, 241, 240]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273586493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[2]R generation[0..12]: [146, 146, 148, 149, 149, 149, 138, 146, 146, 146, 149, 148]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273596822Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273607501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27361673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273626559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Next flush: 242","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273635688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273645018Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"":"downstairs","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273654617Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273664506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"0","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273674095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"1","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273683464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"client":"2","":"downstairs","session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273693153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273702193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"120119d8-c727-4ae0-8115-6cf5c897a6ca is now active with session: 71ae7220-101b-4ce8-859e-5c19c5fad2e9","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273712022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273720591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"session_id":"71ae7220-101b-4ce8-859e-5c19c5fad2e9"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27372956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Activated sub_volume 120119d8-c727-4ae0-8115-6cf5c897a6ca","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27373726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.273745179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.27407377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} 01/30 Write block 113 len 8192 data: 8 8 02/30 Read block 392 len 32768 03/30 Flush 04/30 Write block 119 len 8192 data: 9 10 05/30 Read block 115 len 12288 06/30 Write block 576 len 8192 data: 9 7 07/30 Write block 475 len 4096 data: 15 08/30 Write block 96 len 8192 data: 8 10 09/30 Flush 10/30 Flush 11/30 Flush 12/30 Write block 480 len 36864 data: 11 13 14 14 13 10 12 12 11 13/30 Write block 60 len 16384 data: 9 13 9 11 14/30 Write block 348 len 28672 data: 9 11 14 13 11 9 13 15/30 Write block 11 len 40960 data: 11 10 10 10 13 10 12 12 13 11 16/30 Read block 199 len 36864 17/30 Write block 168 len 4096 data: 6 18/30 Write block 104 len 4096 data: 14 19/30 Flush 20/30 Flush 21/30 Write block 237 len 12288 data: 16 16 14 22/30 Flush 23/30 Write block 550 len 12288 data: 6 6 7 24/30 Read block 461 len 8192 25/30 Read block 326 len 32768 26/30 Write block 544 len 24576 data: 6 7 7 6 7 6 27/30 Write block 559 len 4096 data: 10 28/30 Write block 316 len 12288 data: 14 13 13 29/30 Write block 362 len 4096 data: 14 30/30 Write block 260 len 40960 data: 13 17 14 12 14 17 17 14 14 14 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 2 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 3 Sent Sent Done false 1028 Acked Write 1 Sent Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 3 1 29 Done 2 24 26 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:30.281200489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:30.281400802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1739,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 150 150 239 246 246 F F F <--- 1 020-039 146 150 150 239 246 246 F F F <--- 2 040-059 148 148 148 240 240 240 F F F 3 060-079 149 150 150 241 246 246 F F F <--- 4 080-099 149 150 150 241 243 243 F F F <--- 5 100-119 150 150 150 242 246 246 T F F <--- 6 120-139 138 150 150 226 243 243 T F F <--- 7 140-159 146 146 146 239 239 239 F F F 8 160-179 146 150 150 239 246 246 F F F <--- 9 180-199 146 146 146 239 239 239 F F F 10 200-219 149 149 149 241 241 241 F F F 11 220-239 148 150 150 240 248 248 F F F <--- 12 240-259 149 149 149 241 241 241 F F F 13 260-279 148 148 148 240 240 240 F T T <--- 14 280-299 146 146 146 239 239 239 F F F 15 300-319 148 148 148 240 240 240 F T T <--- 16 320-339 146 146 146 239 239 239 F F F 17 340-359 141 150 150 230 246 246 F F F <--- 18 360-379 144 144 144 236 236 236 F T T <--- 19 380-399 146 146 146 239 239 239 F F F 20 400-419 146 146 146 239 239 239 F F F 21 420-439 148 148 148 240 240 240 F F F 22 440-459 144 144 144 236 236 236 F F F 23 460-479 146 150 150 239 243 243 F F F <--- 24 480-499 145 150 150 238 246 246 F F F <--- 25 500-519 149 149 149 241 241 241 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 144 144 144 236 236 236 F T T <--- 28 560-579 149 150 150 241 243 243 F F F <--- 29 580-599 148 148 148 240 240 240 F F F Max gen: 150, Max flush: 248 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 150 150 150 246 246 246 F F F 2 040-059 148 148 148 240 240 240 F F F 3 060-079 150 150 150 246 246 246 F F F 4 080-099 150 150 150 243 243 243 F F F 5 100-119 150 150 150 246 246 246 F F F 6 120-139 150 150 150 243 243 243 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 150 150 150 246 246 246 F F F 9 180-199 146 146 146 239 239 239 F F F 10 200-219 149 149 149 241 241 241 F F F 11 220-239 150 150 150 248 248 248 F F F 12 240-259 149 149 149 241 241 241 F F F 13 260-279 151 151 151 249 249 249 F F F 14 280-299 146 146 146 239 239 239 F F F 15 300-319 151 151 151 249 249 249 F F F 16 320-339 146 146 146 239 239 239 F F F 17 340-359 150 150 150 246 246 246 F F F 18 360-379 151 151 151 249 249 249 F F F 19 380-399 146 146 146 239 239 239 F F F 20 400-419 146 146 146 239 239 239 F F F 21 420-439 148 148 148 240 240 240 F F F 22 440-459 144 144 144 236 236 236 F F F 23 460-479 150 150 150 243 243 243 F F F 24 480-499 150 150 150 246 246 246 F F F 25 500-519 149 149 149 241 241 241 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 151 151 151 249 249 249 F F F 28 560-579 150 150 150 243 243 243 F F F 29 580-599 148 148 148 240 240 240 F F F Max gen: 151, Max flush: 249 Begin loop 76 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.547521785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:34.55469234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.554746455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.561629296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.561881384Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.561914861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.56192634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.561938179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Crucible 2016e237-7003-45c1-8584-29a02a011028 has session id: d00c71ae-5736-46a9-a6f3-9fce369a53f3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562297867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Upstairs opts: Upstairs UUID: 2016e237-7003-45c1-8584-29a02a011028, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562326175Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Crucible stats registered with UUID: 2016e237-7003-45c1-8584-29a02a011028","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562337164Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:34.562346423Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562466673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562483781Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.56249366Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562643467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562667115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"2016e237-7003-45c1-8584-29a02a011028 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562691713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562700942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562710451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562719871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.56273105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562740989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562759827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:50557)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562787875Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:39272)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562834611Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:56211)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.562926273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563541219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563575226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563632001Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563834784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563853922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[0]R flush_numbers[0..12]: [246, 246, 240, 246, 243, 246, 243, 239, 246, 239, 241, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563913947Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[0]R generation[0..12]: [150, 150, 148, 150, 150, 150, 150, 146, 150, 146, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563925116Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563935835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[1]R flush_numbers[0..12]: [246, 246, 240, 246, 243, 246, 243, 239, 246, 239, 241, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563946224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[1]R generation[0..12]: [150, 150, 148, 150, 150, 150, 150, 146, 150, 146, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563959193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563969952Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[2]R flush_numbers[0..12]: [246, 246, 240, 246, 243, 246, 243, 239, 246, 239, 241, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.563980301Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[2]R generation[0..12]: [150, 150, 148, 150, 150, 150, 150, 146, 150, 146, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.56399057Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564001629Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564011078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564020757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Next flush: 250","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564029807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564038946Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"":"downstairs","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564048685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564057404Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"0","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564067193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"1","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564076772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"client":"2","":"downstairs","session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564086662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564095571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"2016e237-7003-45c1-8584-29a02a011028 is now active with session: d00c71ae-5736-46a9-a6f3-9fce369a53f3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.56410671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564115189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"session_id":"d00c71ae-5736-46a9-a6f3-9fce369a53f3"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564124038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Activated sub_volume 2016e237-7003-45c1-8584-29a02a011028","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564131848Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564139687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.564464419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} 01/30 Write block 391 len 40960 data: 19 25 19 22 19 14 15 17 18 20 02/30 Read block 369 len 28672 03/30 Write block 369 len 28672 data: 17 12 13 12 12 15 15 04/30 Read block 185 len 40960 05/30 Read block 25 len 16384 06/30 Flush 07/30 Write block 447 len 40960 data: 14 15 14 15 14 10 12 11 11 11 08/30 Write block 576 len 4096 data: 10 09/30 Read block 167 len 40960 10/30 Flush 11/30 Write block 130 len 24576 data: 16 18 15 13 13 13 12/30 Read block 420 len 24576 13/30 Write block 187 len 28672 data: 8 6 6 10 11 11 8 14/30 Write block 311 len 40960 data: 11 11 11 12 12 15 14 14 17 16 15/30 Read block 348 len 36864 16/30 Read block 232 len 32768 17/30 Read block 245 len 20480 18/30 Write block 510 len 40960 data: 12 12 15 18 17 15 17 17 15 16 19/30 Read block 301 len 8192 20/30 Write block 34 len 40960 data: 10 12 10 10 8 6 5 5 4 6 21/30 Read block 464 len 4096 22/30 Read block 112 len 24576 23/30 Write block 500 len 40960 data: 7 7 8 9 12 13 16 16 15 13 24/30 Read block 42 len 28672 25/30 Write block 63 len 40960 data: 12 10 10 11 12 12 11 11 11 9 26/30 Write block 280 len 8192 data: 13 13 27/30 Write block 302 len 12288 data: 17 18 18 28/30 Write block 306 len 28672 data: 15 15 13 13 14 12 12 29/30 Write block 105 len 4096 data: 14 30/30 Write block 192 len 36864 data: 12 9 8 9 8 8 7 5 4 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Read 10 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 7 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 2 Sent Sent Done false 1026 Acked Write 3 Sent Sent Sent false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 5 4 38 Done 1 25 26 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1009)) Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:34.573552547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:34.573721202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1747,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 150 150 150 246 246 246 F T T <--- 2 040-059 148 148 148 240 240 240 F T T <--- 3 060-079 150 150 150 246 246 246 F T T <--- 4 080-099 150 150 150 243 243 243 F F F 5 100-119 150 150 150 246 246 246 F T T <--- 6 120-139 150 150 150 243 243 243 F T T <--- 7 140-159 146 146 146 239 239 239 F F F 8 160-179 150 150 150 246 246 246 F F F 9 180-199 146 146 146 239 239 239 F T T <--- 10 200-219 149 149 149 241 241 241 F T T <--- 11 220-239 150 150 150 248 248 248 F F F 12 240-259 149 149 149 241 241 241 F F F 13 260-279 151 151 151 249 249 249 F F F 14 280-299 146 146 146 239 239 239 F T T <--- 15 300-319 151 151 151 249 249 249 F T T <--- 16 320-339 146 146 146 239 239 239 F T T <--- 17 340-359 150 150 150 246 246 246 F F F 18 360-379 151 152 152 249 250 250 F F F <--- 19 380-399 146 152 152 239 250 250 T F F <--- 20 400-419 146 152 152 239 250 250 T F F <--- 21 420-439 148 148 148 240 240 240 F F F 22 440-459 144 152 152 236 251 251 F F F <--- 23 460-479 150 150 150 243 243 243 F F F 24 480-499 150 150 150 246 246 246 F F F 25 500-519 149 149 149 241 241 241 F T T <--- 26 520-539 149 149 149 241 241 241 F F F 27 540-559 151 151 151 249 249 249 F F F 28 560-579 150 152 152 243 251 251 F F F <--- 29 580-599 148 148 148 240 240 240 F F F Max gen: 152, Max flush: 251 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 153 153 153 252 252 252 F F F 3 060-079 153 153 153 252 252 252 F F F 4 080-099 150 150 150 243 243 243 F F F 5 100-119 153 153 153 252 252 252 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 150 150 150 246 246 246 F F F 9 180-199 153 153 153 252 252 252 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 150 150 150 248 248 248 F F F 12 240-259 149 149 149 241 241 241 F F F 13 260-279 151 151 151 249 249 249 F F F 14 280-299 153 153 153 252 252 252 F F F 15 300-319 153 153 153 252 252 252 F F F 16 320-339 153 153 153 252 252 252 F F F 17 340-359 150 150 150 246 246 246 F F F 18 360-379 152 152 152 250 250 250 F F F 19 380-399 152 152 152 250 250 250 F F F 20 400-419 152 152 152 250 250 250 F F F 21 420-439 148 148 148 240 240 240 F F F 22 440-459 152 152 152 251 251 251 F F F 23 460-479 150 150 150 243 243 243 F F F 24 480-499 150 150 150 246 246 246 F F F 25 500-519 153 153 153 252 252 252 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 151 151 151 249 249 249 F F F 28 560-579 152 152 152 251 251 251 F F F 29 580-599 148 148 148 240 240 240 F F F Max gen: 153, Max flush: 252 Begin loop 77 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.851359871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.859715413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.859996969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860030776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860040275Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860051874Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Crucible 84471755-b2ab-411f-935f-baac098bc576 has session id: f4514f58-45ec-40a5-868d-762044643fbd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860407413Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Upstairs opts: Upstairs UUID: 84471755-b2ab-411f-935f-baac098bc576, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.86043774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Crucible stats registered with UUID: 84471755-b2ab-411f-935f-baac098bc576","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860451389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:38.860461148Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860594377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860612815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860623234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860769072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.86078469Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"84471755-b2ab-411f-935f-baac098bc576 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860809438Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860819137Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860829276Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860839315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860849135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860859454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860869663Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:62521)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.860960995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:33276)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861029419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:43104)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861043008Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861636356Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861665024Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861713309Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861951679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.861985566Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[0]R flush_numbers[0..12]: [246, 252, 252, 252, 243, 252, 252, 239, 246, 252, 252, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862059839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[0]R generation[0..12]: [150, 153, 153, 153, 150, 153, 153, 146, 150, 153, 153, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862070828Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862081627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[1]R flush_numbers[0..12]: [246, 252, 252, 252, 243, 252, 252, 239, 246, 252, 252, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862092136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[1]R generation[0..12]: [150, 153, 153, 153, 150, 153, 153, 146, 150, 153, 153, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862102415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862113284Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[2]R flush_numbers[0..12]: [246, 252, 252, 252, 243, 252, 252, 239, 246, 252, 252, 248]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862123694Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[2]R generation[0..12]: [150, 153, 153, 153, 150, 153, 153, 146, 150, 153, 153, 150]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862134083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862146732Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862155941Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.8621655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Next flush: 253","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862174979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862184308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"":"downstairs","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862194077Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862202807Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"0","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862212216Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"1","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862221575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"client":"2","":"downstairs","session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862231124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862240183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"84471755-b2ab-411f-935f-baac098bc576 is now active with session: f4514f58-45ec-40a5-868d-762044643fbd","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862249833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862258452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"session_id":"f4514f58-45ec-40a5-868d-762044643fbd"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862267251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Activated sub_volume 84471755-b2ab-411f-935f-baac098bc576","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.86227507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.86228313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.862594843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} 01/30 Write block 549 len 24576 data: 7 7 7 8 8 9 02/30 Read block 479 len 32768 03/30 Read block 222 len 40960 04/30 Flush 05/30 Write block 461 len 36864 data: 12 13 16 16 14 13 11 10 12 06/30 Read block 17 len 40960 07/30 Read block 212 len 32768 08/30 Read block 288 len 24576 09/30 Write block 543 len 12288 data: 8 7 8 10/30 Read block 213 len 20480 11/30 Read block 536 len 24576 12/30 Write block 508 len 4096 data: 16 13/30 Read block 0 len 32768 14/30 Flush 15/30 Write block 229 len 40960 data: 13 14 15 15 14 16 19 16 17 17 16/30 Read block 191 len 40960 17/30 Read block 161 len 24576 18/30 Read block 251 len 32768 19/30 Flush 20/30 Write block 436 len 28672 data: 11 13 15 13 15 15 17 21/30 Write block 295 len 24576 data: 15 16 15 14 18 14 22/30 Read block 485 len 12288 23/30 Read block 532 len 24576 24/30 Flush 25/30 Write block 74 len 32768 data: 8 9 7 8 8 8 10 11 26/30 Write block 88 len 12288 data: 19 19 16 27/30 Write block 441 len 32768 data: 16 18 14 18 15 16 15 16 28/30 Read block 191 len 12288 29/30 Read block 26 len 12288 30/30 Write block 554 len 32768 data: 10 13 11 10 11 11 9 11 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 8 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Read 10 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:38.87272635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:38.872890066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1755,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 153 153 153 252 252 252 F F F 3 060-079 153 153 153 252 252 252 T T F <--- 4 080-099 150 150 150 243 243 243 T T F <--- 5 100-119 153 153 153 252 252 252 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 150 150 150 246 246 246 F F F 9 180-199 153 153 153 252 252 252 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 154 154 150 255 255 248 F F F <--- 12 240-259 149 149 149 241 241 241 F F F 13 260-279 151 151 151 249 249 249 F F F 14 280-299 154 154 153 256 256 252 F F F <--- 15 300-319 154 154 153 256 256 252 F F F <--- 16 320-339 153 153 153 252 252 252 F F F 17 340-359 150 150 150 246 246 246 F F F 18 360-379 152 152 152 250 250 250 F F F 19 380-399 152 152 152 250 250 250 F F F 20 400-419 152 152 152 250 250 250 F F F 21 420-439 154 154 148 256 256 240 F F F <--- 22 440-459 154 154 152 256 256 251 T T F <--- 23 460-479 154 154 150 254 254 243 F F F <--- 24 480-499 150 150 150 246 246 246 F F F 25 500-519 154 154 153 254 254 252 F F F <--- 26 520-539 149 149 149 241 241 241 F F F 27 540-559 154 154 151 254 254 249 T T T <--- 28 560-579 152 152 152 251 251 251 T T F <--- 29 580-599 148 148 148 240 240 240 F F F Max gen: 154, Max flush: 256 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 153 153 153 252 252 252 F F F 3 060-079 155 155 155 257 257 257 F F F 4 080-099 155 155 155 257 257 257 F F F 5 100-119 153 153 153 252 252 252 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 146 146 146 239 239 239 F F F 8 160-179 150 150 150 246 246 246 F F F 9 180-199 153 153 153 252 252 252 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 154 154 154 255 255 255 F F F 12 240-259 149 149 149 241 241 241 F F F 13 260-279 151 151 151 249 249 249 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 154 154 154 256 256 256 F F F 16 320-339 153 153 153 252 252 252 F F F 17 340-359 150 150 150 246 246 246 F F F 18 360-379 152 152 152 250 250 250 F F F 19 380-399 152 152 152 250 250 250 F F F 20 400-419 152 152 152 250 250 250 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 155 155 155 257 257 257 F F F 23 460-479 154 154 154 254 254 254 F F F 24 480-499 150 150 150 246 246 246 F F F 25 500-519 154 154 154 254 254 254 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 155 155 155 257 257 257 F F F 28 560-579 155 155 155 257 257 257 F F F 29 580-599 148 148 148 240 240 240 F F F Max gen: 155, Max flush: 257 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.12453252Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.132917619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133242821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133268369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133276818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133288067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Crucible d2230500-608c-4353-81fe-6ffdd25c3434 has session id: f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133610439Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Upstairs opts: Upstairs UUID: d2230500-608c-4353-81fe-6ffdd25c3434, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133629697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Crucible stats registered with UUID: d2230500-608c-4353-81fe-6ffdd25c3434","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133639396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:43.133648086Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133779254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133795253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133805202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133962238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133978737Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"d2230500-608c-4353-81fe-6ffdd25c3434 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.133994445Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134003105Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134012564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134021903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134031122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134040741Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134050311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:34455)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134104096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:47783)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134114895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:51845)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134125014Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134784587Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134809464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.134850801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.13508592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135097769Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[0]R flush_numbers[0..12]: [246, 252, 252, 257, 257, 252, 252, 239, 246, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135125757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[0]R generation[0..12]: [150, 153, 153, 155, 155, 153, 153, 146, 150, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135136846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135147895Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[1]R flush_numbers[0..12]: [246, 252, 252, 257, 257, 252, 252, 239, 246, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135158404Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[1]R generation[0..12]: [150, 153, 153, 155, 155, 153, 153, 146, 150, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135168973Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135179682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[2]R flush_numbers[0..12]: [246, 252, 252, 257, 257, 252, 252, 239, 246, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135190031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[2]R generation[0..12]: [150, 153, 153, 155, 155, 153, 153, 146, 150, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.13520041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135211089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135220179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135229658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Next flush: 258","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135238667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135247936Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"":"downstairs","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135258865Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135267564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"0","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135277134Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"1","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135286453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"client":"2","":"downstairs","session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135296212Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135305191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"d2230500-608c-4353-81fe-6ffdd25c3434 is now active with session: f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.13531495Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.13532355Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"session_id":"f8b53ec1-86d4-4e68-b50e-c876a8d2cdfb"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135332579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Activated sub_volume d2230500-608c-4353-81fe-6ffdd25c3434","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135340588Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135348417Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.135680149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} 01/30 Write block 159 len 32768 data: 13 12 9 10 8 7 8 6 02/30 Read block 68 len 36864 03/30 Write block 502 len 8192 data: 9 10 04/30 Write block 570 len 4096 data: 9 05/30 Write block 152 len 40960 data: 10 16 16 16 14 15 14 14 13 10 06/30 Write block 394 len 40960 data: 23 20 15 16 18 19 21 23 22 18 07/30 Write block 324 len 20480 data: 14 14 14 13 13 08/30 Read block 473 len 40960 09/30 Write block 446 len 28672 data: 17 16 17 15 16 15 11 10/30 Write block 547 len 24576 data: 7 8 8 8 8 9 11/30 Flush 12/30 Write block 340 len 20480 data: 14 14 13 13 12 13/30 Read block 280 len 36864 14/30 Read block 431 len 20480 15/30 Write block 337 len 12288 data: 14 17 15 16/30 Read block 73 len 36864 17/30 Write block 441 len 12288 data: 17 19 15 18/30 Write block 265 len 28672 data: 18 18 15 15 15 16 16 19/30 Read block 282 len 20480 20/30 Write block 77 len 36864 data: 9 9 9 11 12 9 10 11 13 21/30 Read block 465 len 4096 22/30 Write block 255 len 28672 data: 10 10 10 13 13 14 18 23/30 Read block 276 len 16384 24/30 Write block 474 len 12288 data: 14 16 12 25/30 Write block 360 len 12288 data: 10 15 15 26/30 Read block 318 len 20480 27/30 Write block 553 len 20480 data: 9 11 14 12 11 28/30 Write block 452 len 16384 data: 12 13 12 12 29/30 Write block 589 len 40960 data: 16 17 18 16 12 8 8 8 7 6 30/30 Read block 570 len 36864 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 10 Done Done Done false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 5 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 10 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 22 23 Done 30 29 8 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:43.144297418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:43.144477542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1763,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 153 153 153 252 252 252 F F F 3 060-079 155 155 155 257 257 257 T T F <--- 4 080-099 155 155 155 257 257 257 T T F <--- 5 100-119 153 153 153 252 252 252 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 146 258 258 239 F F T <--- 8 160-179 156 156 150 258 258 246 F F T <--- 9 180-199 153 153 153 252 252 252 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 154 154 154 255 255 255 F F F 12 240-259 149 149 149 241 241 241 T T F <--- 13 260-279 151 151 151 249 249 249 T T F <--- 14 280-299 154 154 154 256 256 256 F F F 15 300-319 154 154 154 256 256 256 F F F 16 320-339 156 156 153 258 258 252 T T T <--- 17 340-359 150 150 150 246 246 246 T T F <--- 18 360-379 152 152 152 250 250 250 T T F <--- 19 380-399 156 156 152 258 258 250 F F T <--- 20 400-419 156 156 152 258 258 250 F F T <--- 21 420-439 154 154 154 256 256 256 F F F 22 440-459 156 156 155 258 258 257 T T F <--- 23 460-479 154 154 154 254 254 254 T T F <--- 24 480-499 150 150 150 246 246 246 F F F 25 500-519 156 156 154 258 258 254 F F T <--- 26 520-539 149 149 149 241 241 241 F F F 27 540-559 156 156 155 258 258 257 T T F <--- 28 560-579 156 156 155 258 258 257 F F T <--- 29 580-599 148 148 148 240 240 240 T T F <--- Max gen: 156, Max flush: 258 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 153 153 153 252 252 252 F F F 3 060-079 157 157 157 259 259 259 F F F 4 080-099 157 157 157 259 259 259 F F F 5 100-119 153 153 153 252 252 252 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 156 258 258 258 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 153 153 153 252 252 252 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 154 154 154 255 255 255 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 157 157 157 259 259 259 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 154 154 154 256 256 256 F F F 16 320-339 157 157 157 259 259 259 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 157 157 157 259 259 259 F F F 19 380-399 156 156 156 258 258 258 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 157 157 157 259 259 259 F F F 23 460-479 157 157 157 259 259 259 F F F 24 480-499 150 150 150 246 246 246 F F F 25 500-519 156 156 156 258 258 258 F F F 26 520-539 149 149 149 241 241 241 F F F 27 540-559 157 157 157 259 259 259 F F F 28 560-579 156 156 156 258 258 258 F F F 29 580-599 157 157 157 259 259 259 F F F Max gen: 157, Max flush: 259 Begin loop 79 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.428269185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.436454632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.436758246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.436792263Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.436803562Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.436814961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Crucible 278ce790-0e63-4687-9018-b0b8f385988a has session id: 2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437114305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Upstairs opts: Upstairs UUID: 278ce790-0e63-4687-9018-b0b8f385988a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437142372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Crucible stats registered with UUID: 278ce790-0e63-4687-9018-b0b8f385988a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437153261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:47.437162381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437306468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437335346Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437346085Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43751471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437524879Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"278ce790-0e63-4687-9018-b0b8f385988a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437541188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437549907Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437559516Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437571015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437580514Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437616851Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437634339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"ds_connection connected from Ok(127.0.0.1:52237)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437644959Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"ds_connection connected from Ok(127.0.0.1:32810)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.437655238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"ds_connection connected from Ok(127.0.0.1:51319)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43774054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438263575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438276314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43831772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438584887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438594206Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[0]R flush_numbers[0..12]: [246, 252, 252, 259, 259, 252, 252, 258, 258, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438619524Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[0]R generation[0..12]: [150, 153, 153, 157, 157, 153, 153, 156, 156, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438630813Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438641782Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[1]R flush_numbers[0..12]: [246, 252, 252, 259, 259, 252, 252, 258, 258, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438654281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[1]R generation[0..12]: [150, 153, 153, 157, 157, 153, 153, 156, 156, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43866473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438675289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[2]R flush_numbers[0..12]: [246, 252, 252, 259, 259, 252, 252, 258, 258, 252, 252, 255]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438685728Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[2]R generation[0..12]: [150, 153, 153, 157, 157, 153, 153, 156, 156, 153, 153, 154]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438695997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438706666Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438715985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438725534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Next flush: 260","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438734614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438743813Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"":"downstairs","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438753602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438762271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"0","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43877204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"1","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.43878145Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"client":"2","":"downstairs","session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438791059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438801078Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"278ce790-0e63-4687-9018-b0b8f385988a is now active with session: 2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438810977Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438819636Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"session_id":"2b078fd8-5b6e-4a9c-934c-4fc56b6a1f09"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438828625Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Activated sub_volume 278ce790-0e63-4687-9018-b0b8f385988a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438836655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.438844764Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.439173835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} 01/30 Write block 226 len 32768 data: 10 11 14 14 15 16 16 15 02/30 Flush 03/30 Write block 305 len 16384 data: 16 16 16 14 04/30 Read block 584 len 32768 05/30 Read block 360 len 16384 06/30 Read block 192 len 36864 07/30 Write block 111 len 20480 data: 8 7 9 9 9 08/30 Read block 273 len 20480 09/30 Read block 276 len 12288 10/30 Write block 191 len 8192 data: 12 13 11/30 Read block 268 len 36864 12/30 Read block 507 len 40960 13/30 Read block 188 len 24576 14/30 Read block 50 len 40960 15/30 Read block 429 len 36864 16/30 Write block 446 len 12288 data: 18 17 18 17/30 Write block 231 len 8192 data: 17 17 18/30 Write block 566 len 24576 data: 9 7 7 11 10 11 19/30 Write block 455 len 40960 data: 13 12 9 13 11 12 13 14 17 17 20/30 Flush 21/30 Write block 184 len 4096 data: 11 22/30 Write block 52 len 12288 data: 14 13 16 23/30 Flush 24/30 Flush 25/30 Write block 534 len 28672 data: 14 13 13 13 11 9 10 26/30 Read block 550 len 4096 27/30 Read block 452 len 28672 28/30 Read block 196 len 24576 29/30 Write block 378 len 24576 data: 13 12 13 15 14 17 30/30 Write block 494 len 4096 data: 6 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:47.447817102Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:47.447974579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1771,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 158 153 158 262 252 262 F F F <--- 3 060-079 157 157 157 259 259 259 F F F 4 080-099 157 157 157 259 259 259 F F F 5 100-119 158 153 158 261 252 261 F F F <--- 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 156 258 258 258 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 158 153 158 262 252 262 F F F <--- 10 200-219 153 153 153 252 252 252 F F F 11 220-239 158 154 158 261 255 261 F T F <--- 12 240-259 157 157 157 259 259 259 F F F 13 260-279 157 157 157 259 259 259 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 154 158 261 256 261 F F F <--- 16 320-339 157 157 157 259 259 259 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 157 157 157 259 259 259 T F T <--- 19 380-399 156 156 156 258 258 258 T F T <--- 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 158 157 158 261 259 261 F F F <--- 23 460-479 158 157 158 261 259 261 F F F <--- 24 480-499 150 150 150 246 246 246 T F T <--- 25 500-519 156 156 156 258 258 258 F F F 26 520-539 149 149 149 241 241 241 T F T <--- 27 540-559 157 157 157 259 259 259 T F T <--- 28 560-579 158 156 158 261 258 261 F F F <--- 29 580-599 157 157 157 259 259 259 F F F Max gen: 158, Max flush: 262 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 246 246 246 F F F 1 020-039 153 153 153 252 252 252 F F F 2 040-059 158 158 158 262 262 262 F F F 3 060-079 157 157 157 259 259 259 F F F 4 080-099 157 157 157 259 259 259 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 156 258 258 258 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 158 158 158 262 262 262 F F F 10 200-219 153 153 153 252 252 252 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 157 157 157 259 259 259 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 157 157 157 259 259 259 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 159 159 159 263 263 263 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 158 158 158 261 261 261 F F F 23 460-479 158 158 158 261 261 261 F F F 24 480-499 159 159 159 263 263 263 F F F 25 500-519 156 156 156 258 258 258 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 157 157 157 259 259 259 F F F Max gen: 159, Max flush: 263 Begin loop 80 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.703718555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.711857556Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71215558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712185118Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712193667Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712205176Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Crucible 5b106d00-a2a1-4646-add5-251c6577f43a has session id: ff302e24-9a2f-459f-9bfb-8253248839b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712538527Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Upstairs opts: Upstairs UUID: 5b106d00-a2a1-4646-add5-251c6577f43a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712601822Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Crucible stats registered with UUID: 5b106d00-a2a1-4646-add5-251c6577f43a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712629139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:51.712638968Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712761818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712778406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712788335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712927103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712943662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"5b106d00-a2a1-4646-add5-251c6577f43a active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71296683Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712976139Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712985568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.712995027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713004466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713014236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713023975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:34040)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713089519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37765)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713114767Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:33654)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713127976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71376265Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713782839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.713811326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714077153Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71411326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[0]R flush_numbers[0..12]: [246, 252, 262, 259, 259, 261, 252, 258, 258, 262, 252, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714181724Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[0]R generation[0..12]: [150, 153, 158, 157, 157, 158, 153, 156, 156, 158, 153, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714193003Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714203852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[1]R flush_numbers[0..12]: [246, 252, 262, 259, 259, 261, 252, 258, 258, 262, 252, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714214261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[1]R generation[0..12]: [150, 153, 158, 157, 157, 158, 153, 156, 156, 158, 153, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71422461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714235279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[2]R flush_numbers[0..12]: [246, 252, 262, 259, 259, 261, 252, 258, 258, 262, 252, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714245658Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[2]R generation[0..12]: [150, 153, 158, 157, 157, 158, 153, 156, 156, 158, 153, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714256117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714269536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714278755Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714288215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Next flush: 264","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714297474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714306673Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"":"downstairs","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714316232Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714324871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"0","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714334341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"1","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.71434382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"client":"2","":"downstairs","session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714353449Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714362358Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"5b106d00-a2a1-4646-add5-251c6577f43a is now active with session: ff302e24-9a2f-459f-9bfb-8253248839b5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714372107Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714380647Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"session_id":"ff302e24-9a2f-459f-9bfb-8253248839b5"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714389456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Activated sub_volume 5b106d00-a2a1-4646-add5-251c6577f43a","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714397325Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-09-18T03:33:51.714405294Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.714678171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} 01/30 Write block 321 len 20480 data: 13 15 15 15 15 02/30 Read block 81 len 8192 03/30 Write block 11 len 24576 data: 12 11 11 11 14 11 04/30 Read block 319 len 8192 05/30 Read block 506 len 36864 06/30 Read block 197 len 24576 07/30 Read block 340 len 36864 08/30 Read block 59 len 20480 09/30 Write block 74 len 32768 data: 9 10 8 10 10 10 12 13 10/30 Read block 446 len 32768 11/30 Read block 380 len 24576 12/30 Write block 59 len 12288 data: 10 10 14 13/30 Read block 126 len 20480 14/30 Write block 263 len 16384 data: 13 15 19 19 15/30 Write block 26 len 4096 data: 12 16/30 Write block 449 len 20480 data: 16 17 16 13 14 17/30 Flush 18/30 Write block 391 len 32768 data: 20 26 20 24 21 16 17 19 19/30 Read block 355 len 40960 20/30 Read block 384 len 12288 21/30 Flush 22/30 Write block 323 len 4096 data: 16 23/30 Flush 24/30 Write block 195 len 40960 data: 10 9 9 8 6 5 5 6 6 7 25/30 Flush 26/30 Write block 593 len 24576 data: 13 9 9 9 8 7 27/30 Read block 546 len 32768 28/30 Write block 64 len 20480 data: 11 11 12 13 13 29/30 Read block 159 len 4096 30/30 Read block 32 len 36864 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 24 25 Done 30 29 6 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:51.723815035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:51.723989179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1779,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 150 264 264 246 F F T <--- 1 020-039 160 160 153 264 264 252 F F F <--- 2 040-059 160 160 158 264 264 262 F F F <--- 3 060-079 160 160 157 264 264 259 T T F <--- 4 080-099 160 160 157 264 264 259 F F F <--- 5 100-119 158 158 158 261 261 261 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 156 258 258 258 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 160 160 158 267 267 262 F F F <--- 10 200-219 160 160 153 267 267 252 F F F <--- 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 157 264 264 259 F F F <--- 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 160 160 157 266 266 259 F F T <--- 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 159 265 265 263 F F F <--- 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 158 264 264 261 F F F <--- 23 460-479 158 158 158 261 261 261 F F F 24 480-499 159 159 159 263 263 263 F F F 25 500-519 156 156 156 258 258 258 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 157 157 157 259 259 259 T T F <--- Max gen: 160, Max flush: 267 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 264 264 264 F F F 1 020-039 160 160 160 264 264 264 F F F 2 040-059 160 160 160 264 264 264 F F F 3 060-079 161 161 161 268 268 268 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 158 158 158 261 261 261 F F F 6 120-139 153 153 153 252 252 252 F F F 7 140-159 156 156 156 258 258 258 F F F 8 160-179 156 156 156 258 258 258 F F F 9 180-199 160 160 160 267 267 267 F F F 10 200-219 160 160 160 267 267 267 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 160 160 160 266 266 266 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 160 265 265 265 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 158 158 158 261 261 261 F F F 24 480-499 159 159 159 263 263 263 F F F 25 500-519 156 156 156 258 258 258 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 161 161 161 268 268 268 F F F Max gen: 161, Max flush: 268 Begin loop 81 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.977320985Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:55.984478472Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.984533047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.991513939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.991765297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.991800214Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.991809503Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.991821112Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Crucible 538ee404-4e2a-4637-82b2-398de7693ee7 has session id: 3725cba6-a07d-4072-a5c8-5f25ba923eb2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992169422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Upstairs opts: Upstairs UUID: 538ee404-4e2a-4637-82b2-398de7693ee7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992201389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Crucible stats registered with UUID: 538ee404-4e2a-4637-82b2-398de7693ee7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992211638Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:55.992220517Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992344876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992360765Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992372584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992541289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992564677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"538ee404-4e2a-4637-82b2-398de7693ee7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992590995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992600054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992609623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992619062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992628392Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992638041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992662819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:35274)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992675927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:45744)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992702395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:49121)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.992716224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"io task","client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993288064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993365587Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993434701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.99356068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993613566Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[0]R flush_numbers[0..12]: [264, 264, 264, 268, 264, 261, 252, 258, 258, 267, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993726076Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[0]R generation[0..12]: [160, 160, 160, 161, 160, 158, 153, 156, 156, 160, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993752104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993763893Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[1]R flush_numbers[0..12]: [264, 264, 264, 268, 264, 261, 252, 258, 258, 267, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993774602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[1]R generation[0..12]: [160, 160, 160, 161, 160, 158, 153, 156, 156, 160, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993785061Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.99379584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[2]R flush_numbers[0..12]: [264, 264, 264, 268, 264, 261, 252, 258, 258, 267, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993806549Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[2]R generation[0..12]: [160, 160, 160, 161, 160, 158, 153, 156, 156, 160, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993816978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993827627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993836856Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993846405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Next flush: 269","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993858184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993867564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"":"downstairs","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993877193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993885892Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"0","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993895531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"1","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.99390493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"client":"2","":"downstairs","session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.99391485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993923709Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"538ee404-4e2a-4637-82b2-398de7693ee7 is now active with session: 3725cba6-a07d-4072-a5c8-5f25ba923eb2","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993933448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993942007Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"session_id":"3725cba6-a07d-4072-a5c8-5f25ba923eb2"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993950876Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Activated sub_volume 538ee404-4e2a-4637-82b2-398de7693ee7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993958646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.993966505Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:55.994273328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} 01/30 Write block 105 len 16384 data: 15 11 11 9 02/30 Write block 482 len 32768 data: 15 15 14 11 13 13 12 10 03/30 Read block 103 len 4096 04/30 Read block 446 len 24576 05/30 Read block 20 len 36864 06/30 Write block 42 len 40960 data: 5 7 12 11 13 13 16 15 16 15 07/30 Read block 159 len 8192 08/30 Write block 169 len 20480 data: 6 8 9 9 7 09/30 Read block 243 len 32768 10/30 Write block 587 len 4096 data: 15 11/30 Write block 191 len 28672 data: 13 14 10 9 11 10 10 12/30 Flush 13/30 Flush 14/30 Flush 15/30 Flush 16/30 Write block 107 len 28672 data: 12 10 7 10 9 8 10 17/30 Read block 216 len 20480 18/30 Write block 154 len 16384 data: 17 17 15 16 19/30 Read block 553 len 20480 20/30 Flush 21/30 Write block 337 len 12288 data: 15 18 16 22/30 Write block 515 len 20480 data: 16 18 18 16 17 23/30 Read block 246 len 16384 24/30 Write block 132 len 4096 data: 16 25/30 Write block 157 len 8192 data: 17 15 26/30 Read block 55 len 32768 27/30 Read block 312 len 20480 28/30 Write block 32 len 16384 data: 9 11 11 13 29/30 Read block 318 len 16384 30/30 Write block 70 len 12288 data: 12 12 10 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 1 26 Done 6 29 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:33:56.002941183Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:33:56.003146135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1787,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 264 264 264 F F F 1 020-039 160 160 160 264 264 264 F T T <--- 2 040-059 160 162 162 264 269 269 T F F <--- 3 060-079 161 161 161 268 268 268 F T T <--- 4 080-099 160 160 160 264 264 264 F F F 5 100-119 158 162 162 261 273 273 T F F <--- 6 120-139 153 153 153 252 252 252 F T T <--- 7 140-159 156 162 162 258 273 273 F T T <--- 8 160-179 156 162 162 258 269 269 F F F <--- 9 180-199 160 162 162 267 269 269 F F F <--- 10 200-219 160 160 160 267 267 267 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 160 160 160 266 266 266 F T T <--- 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 160 265 265 265 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 158 158 158 261 261 261 F F F 24 480-499 159 162 162 263 269 269 T F F <--- 25 500-519 156 156 156 258 258 258 F T T <--- 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 161 162 162 268 269 269 F F F <--- Max gen: 162, Max flush: 273 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 264 264 264 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 162 162 162 269 269 269 F F F 3 060-079 163 163 163 274 274 274 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 162 162 162 273 273 273 F F F 6 120-139 163 163 163 274 274 274 F F F 7 140-159 163 163 163 274 274 274 F F F 8 160-179 162 162 162 269 269 269 F F F 9 180-199 162 162 162 269 269 269 F F F 10 200-219 160 160 160 267 267 267 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 154 154 154 256 256 256 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 163 163 163 274 274 274 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 160 265 265 265 F F F 20 400-419 156 156 156 258 258 258 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 158 158 158 261 261 261 F F F 24 480-499 162 162 162 269 269 269 F F F 25 500-519 163 163 163 274 274 274 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 162 162 162 269 269 269 F F F Max gen: 163, Max flush: 274 Begin loop 82 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.262948018Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.271433149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.271741432Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.271776799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.271788138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.271799827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Crucible 23cffbd5-bf92-4c0e-b24c-c736d7a88373 has session id: 67e93047-6957-4e52-91aa-b99fd55a69df","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272117759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Upstairs opts: Upstairs UUID: 23cffbd5-bf92-4c0e-b24c-c736d7a88373, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272147856Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Crucible stats registered with UUID: 23cffbd5-bf92-4c0e-b24c-c736d7a88373","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272158905Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:00.272168005Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272308662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272327821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.27233787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272484087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272500546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"23cffbd5-bf92-4c0e-b24c-c736d7a88373 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272524224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272533313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272542702Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272553851Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.27256327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272573219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272582898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"ds_connection connected from Ok(127.0.0.1:61642)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272669091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"ds_connection connected from Ok(127.0.0.1:62806)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272735025Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"ds_connection connected from Ok(127.0.0.1:39261)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.272759423Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"io task","client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273324764Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273343372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273406297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273678323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273704421Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[0]R flush_numbers[0..12]: [264, 274, 269, 274, 264, 273, 274, 274, 269, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273763536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[0]R generation[0..12]: [160, 163, 162, 163, 160, 162, 163, 163, 162, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273775245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273786104Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[1]R flush_numbers[0..12]: [264, 274, 269, 274, 264, 273, 274, 274, 269, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273799382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[1]R generation[0..12]: [160, 163, 162, 163, 160, 162, 163, 163, 162, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273809822Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273820601Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[2]R flush_numbers[0..12]: [264, 274, 269, 274, 264, 273, 274, 274, 269, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.27383101Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[2]R generation[0..12]: [160, 163, 162, 163, 160, 162, 163, 163, 162, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273841629Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273852508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273861887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273871626Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Next flush: 275","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273880805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273890025Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"":"downstairs","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273899734Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273908213Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"0","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273917632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"1","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273926931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"client":"2","":"downstairs","session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.27393779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.27394678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"23cffbd5-bf92-4c0e-b24c-c736d7a88373 is now active with session: 67e93047-6957-4e52-91aa-b99fd55a69df","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273956529Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273965068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"session_id":"67e93047-6957-4e52-91aa-b99fd55a69df"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273973847Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Activated sub_volume 23cffbd5-bf92-4c0e-b24c-c736d7a88373","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273981607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.273989446Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.274353084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} 01/30 Write block 169 len 40960 data: 7 9 10 10 8 7 8 11 11 9 02/30 Read block 119 len 4096 03/30 Read block 142 len 4096 04/30 Read block 236 len 28672 05/30 Read block 154 len 8192 06/30 Read block 588 len 32768 07/30 Flush 08/30 Write block 474 len 20480 data: 15 17 13 12 15 09/30 Write block 322 len 36864 data: 16 17 16 16 15 14 14 11 11 10/30 Write block 171 len 24576 data: 11 11 9 8 9 12 11/30 Read block 409 len 32768 12/30 Read block 261 len 40960 13/30 Read block 555 len 40960 14/30 Read block 213 len 32768 15/30 Read block 493 len 28672 16/30 Read block 190 len 28672 17/30 Flush 18/30 Write block 130 len 32768 data: 17 19 17 14 14 14 15 14 19/30 Write block 280 len 4096 data: 14 20/30 Write block 400 len 36864 data: 22 24 23 19 19 19 17 17 16 21/30 Write block 42 len 40960 data: 6 8 13 12 14 14 17 16 17 16 22/30 Read block 117 len 8192 23/30 Write block 465 len 4096 data: 15 24/30 Read block 528 len 16384 25/30 Read block 372 len 36864 26/30 Read block 562 len 40960 27/30 Write block 113 len 36864 data: 11 10 10 10 8 10 10 11 9 28/30 Read block 92 len 4096 29/30 Read block 308 len 20480 30/30 Read block 457 len 12288 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 30 5 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:00.28312372Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:00.283304844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1795,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 264 264 264 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 162 162 162 269 269 269 T F T <--- 3 060-079 163 163 163 274 274 274 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 162 162 162 273 273 273 T F T <--- 6 120-139 163 163 163 274 274 274 T F T <--- 7 140-159 163 163 163 274 274 274 F F F 8 160-179 164 162 164 276 269 276 F T F <--- 9 180-199 162 162 162 269 269 269 F F F 10 200-219 160 160 160 267 267 267 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 154 154 154 256 256 256 T F T <--- 15 300-319 158 158 158 261 261 261 F F F 16 320-339 164 163 164 276 274 276 F F F <--- 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 160 265 265 265 F F F 20 400-419 156 156 156 258 258 258 T F T <--- 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 164 158 164 276 261 276 T F T <--- 24 480-499 162 162 162 269 269 269 F F F 25 500-519 163 163 163 274 274 274 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 162 162 162 269 269 269 F F F Max gen: 164, Max flush: 276 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 264 264 264 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 165 165 165 277 277 277 F F F 3 060-079 163 163 163 274 274 274 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 165 165 165 277 277 277 F F F 7 140-159 163 163 163 274 274 274 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 162 162 162 269 269 269 F F F 10 200-219 160 160 160 267 267 267 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 158 158 158 261 261 261 F F F 16 320-339 164 164 164 276 276 276 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 F F F 19 380-399 160 160 160 265 265 265 F F F 20 400-419 165 165 165 277 277 277 F F F 21 420-439 154 154 154 256 256 256 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 165 165 165 277 277 277 F F F 24 480-499 162 162 162 269 269 269 F F F 25 500-519 163 163 163 274 274 274 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 159 159 159 263 263 263 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 162 162 162 269 269 269 F F F Max gen: 165, Max flush: 277 Begin loop 83 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.527898922Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.535978938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536264624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536288471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536297001Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53630819Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Crucible 23c3c268-fdf3-450b-8a1e-8c5301a074ea has session id: 8e6c0996-962f-4e9b-8794-f0c32e2ea9ac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536591345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Upstairs opts: Upstairs UUID: 23c3c268-fdf3-450b-8a1e-8c5301a074ea, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536634791Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Crucible stats registered with UUID: 23c3c268-fdf3-450b-8a1e-8c5301a074ea","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53664733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:04.536656249Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536790958Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536844603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536865761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.536975992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53698959Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"23c3c268-fdf3-450b-8a1e-8c5301a074ea active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537043576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537052965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537062534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537072123Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537081292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537090882Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537100571Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:64392)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537180414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:38713)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537205022Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:35296)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537297494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"io task","client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537929888Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537971835Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.537983874Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538242621Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53826204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[0]R flush_numbers[0..12]: [264, 274, 277, 274, 264, 277, 277, 274, 276, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538314065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[0]R generation[0..12]: [160, 163, 165, 163, 160, 165, 165, 163, 164, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538325034Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538335803Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[1]R flush_numbers[0..12]: [264, 274, 277, 274, 264, 277, 277, 274, 276, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538346122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[1]R generation[0..12]: [160, 163, 165, 163, 160, 165, 165, 163, 164, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538356471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53836699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[2]R flush_numbers[0..12]: [264, 274, 277, 274, 264, 277, 277, 274, 276, 269, 267, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538377389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[2]R generation[0..12]: [160, 163, 165, 163, 160, 165, 165, 163, 164, 162, 160, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538389998Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538400687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538409877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538419456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Next flush: 278","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538428555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538437854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"":"downstairs","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538447353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538455993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"0","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538465352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"1","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538474851Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"client":"2","":"downstairs","session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.53848436Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538493289Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"23c3c268-fdf3-450b-8a1e-8c5301a074ea is now active with session: 8e6c0996-962f-4e9b-8794-f0c32e2ea9ac","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538502939Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538511508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"session_id":"8e6c0996-962f-4e9b-8794-f0c32e2ea9ac"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538520467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Activated sub_volume 23c3c268-fdf3-450b-8a1e-8c5301a074ea","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538528326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538536416Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.538863537Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} 01/30 Flush 02/30 Write block 403 len 28672 data: 20 20 20 18 18 17 13 03/30 Write block 46 len 40960 data: 15 15 18 17 18 17 15 14 17 14 04/30 Write block 429 len 28672 data: 13 14 12 12 10 11 11 05/30 Write block 546 len 8192 data: 8 8 06/30 Write block 74 len 12288 data: 10 11 9 07/30 Read block 87 len 40960 08/30 Read block 320 len 12288 09/30 Write block 311 len 28672 data: 13 13 12 13 13 16 15 10/30 Write block 593 len 16384 data: 14 10 10 10 11/30 Write block 16 len 4096 data: 12 12/30 Read block 80 len 8192 13/30 Read block 368 len 20480 14/30 Read block 502 len 20480 15/30 Flush 16/30 Write block 328 len 28672 data: 15 12 12 10 8 9 10 17/30 Read block 85 len 24576 18/30 Read block 45 len 40960 19/30 Read block 56 len 28672 20/30 Read block 545 len 36864 21/30 Flush 22/30 Write block 517 len 12288 data: 19 17 18 23/30 Read block 92 len 8192 24/30 Read block 351 len 20480 25/30 Write block 193 len 40960 data: 11 10 12 11 11 9 7 6 6 7 26/30 Read block 239 len 32768 27/30 Read block 438 len 32768 28/30 Read block 148 len 36864 29/30 Write block 378 len 32768 data: 14 13 14 16 15 18 14 14 30/30 Write block 7 len 8192 data: 12 13 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Done Done Done false 1002 Acked Write 10 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 27 27 1 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:04.547578978Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:04.547748933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1803,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 166 166 160 279 279 264 T T F <--- 1 020-039 163 163 163 274 274 274 F F F 2 040-059 166 166 165 279 279 277 F F F <--- 3 060-079 166 166 163 279 279 274 F F F <--- 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 165 165 165 277 277 277 F F F 7 140-159 163 163 163 274 274 274 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 162 162 162 269 269 269 T T F <--- 10 200-219 160 160 160 267 267 267 T T F <--- 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 166 166 158 279 279 261 F F F <--- 16 320-339 166 166 164 280 280 276 F F F <--- 17 340-359 157 157 157 259 259 259 F F F 18 360-379 159 159 159 263 263 263 T T F <--- 19 380-399 160 160 160 265 265 265 T T F <--- 20 400-419 166 166 165 279 279 277 F F T <--- 21 420-439 166 166 154 279 279 256 F F F <--- 22 440-459 160 160 160 264 264 264 F F F 23 460-479 165 165 165 277 277 277 F F F 24 480-499 162 162 162 269 269 269 F F F 25 500-519 163 163 163 274 274 274 T T F <--- 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 159 279 279 263 F F F <--- 28 560-579 158 158 158 261 261 261 F F F 29 580-599 166 166 162 279 279 269 F F F <--- Max gen: 166, Max flush: 280 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 281 281 281 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 166 166 166 279 279 279 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 165 165 165 277 277 277 F F F 7 140-159 163 163 163 274 274 274 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 167 167 167 281 281 281 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 166 166 166 279 279 279 F F F 16 320-339 166 166 166 280 280 280 F F F 17 340-359 157 157 157 259 259 259 F F F 18 360-379 167 167 167 281 281 281 F F F 19 380-399 167 167 167 281 281 281 F F F 20 400-419 166 166 166 279 279 279 F F F 21 420-439 166 166 166 279 279 279 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 165 165 165 277 277 277 F F F 24 480-499 162 162 162 269 269 269 F F F 25 500-519 167 167 167 281 281 281 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 158 158 158 261 261 261 F F F 29 580-599 166 166 166 279 279 279 F F F Max gen: 167, Max flush: 281 Begin loop 84 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.805090831Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813209013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813489839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813521226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813529595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813540914Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Crucible cc53f637-1cf4-45ac-b920-6dbc6708b31c has session id: 991a5636-fd4d-4018-842d-049c4b895243","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813892624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Upstairs opts: Upstairs UUID: cc53f637-1cf4-45ac-b920-6dbc6708b31c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.813921261Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Crucible stats registered with UUID: cc53f637-1cf4-45ac-b920-6dbc6708b31c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.81393253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:08.813942019Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814060519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814077468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814087497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814257202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.81428102Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"cc53f637-1cf4-45ac-b920-6dbc6708b31c active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814297059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814305598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814315087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814324396Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814333805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814343405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814380271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"ds_connection connected from Ok(127.0.0.1:49644)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814445136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"ds_connection connected from Ok(127.0.0.1:58282)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814456185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"ds_connection connected from Ok(127.0.0.1:35026)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.814466284Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"io task","client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.8151979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815230057Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815246326Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815500344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815528501Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[0]R flush_numbers[0..12]: [281, 274, 279, 279, 264, 277, 277, 274, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815593046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[0]R generation[0..12]: [167, 163, 166, 166, 160, 165, 165, 163, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815604255Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815615204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[1]R flush_numbers[0..12]: [281, 274, 279, 279, 264, 277, 277, 274, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815625713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[1]R generation[0..12]: [167, 163, 166, 166, 160, 165, 165, 163, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815636182Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815646901Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[2]R flush_numbers[0..12]: [281, 274, 279, 279, 264, 277, 277, 274, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.81565716Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[2]R generation[0..12]: [167, 163, 166, 166, 160, 165, 165, 163, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815667499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815678228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815687267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815696787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815705896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815715065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"":"downstairs","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815725994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815734563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"0","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815744062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"1","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815753452Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"client":"2","":"downstairs","session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815763181Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.81577208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"cc53f637-1cf4-45ac-b920-6dbc6708b31c is now active with session: 991a5636-fd4d-4018-842d-049c4b895243","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815781799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815790378Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"session_id":"991a5636-fd4d-4018-842d-049c4b895243"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815799448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Activated sub_volume cc53f637-1cf4-45ac-b920-6dbc6708b31c","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815807347Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.815815246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.81611415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} 01/30 Write block 135 len 28672 data: 15 16 15 11 12 9 9 02/30 Write block 577 len 24576 data: 8 5 5 6 5 7 03/30 Read block 434 len 36864 04/30 Read block 501 len 8192 05/30 Read block 393 len 20480 06/30 Read block 389 len 24576 07/30 Write block 47 len 8192 data: 16 19 08/30 Read block 199 len 20480 09/30 Read block 235 len 16384 10/30 Flush 11/30 Write block 306 len 24576 data: 17 17 15 14 15 14 12/30 Flush 13/30 Write block 478 len 20480 data: 16 13 12 14 16 14/30 Write block 244 len 32768 data: 9 9 11 14 14 13 9 9 15/30 Read block 60 len 28672 16/30 Write block 333 len 40960 data: 10 11 10 12 16 19 17 15 15 14 17/30 Read block 309 len 32768 18/30 Write block 142 len 32768 data: 9 9 11 12 11 10 9 12 19/30 Read block 53 len 40960 20/30 Write block 379 len 40960 data: 14 15 17 16 19 15 15 17 21 20 21/30 Read block 292 len 36864 22/30 Write block 431 len 36864 data: 13 13 11 12 12 12 14 16 14 23/30 Read block 109 len 12288 24/30 Read block 574 len 40960 25/30 Read block 468 len 28672 26/30 Write block 562 len 8192 data: 11 8 27/30 Write block 400 len 40960 data: 23 25 24 21 21 21 19 19 18 14 28/30 Read block 203 len 28672 29/30 Read block 535 len 12288 30/30 Write block 401 len 28672 data: 26 25 22 22 22 20 20 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1010 Acked Write 6 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 20 22 Done 19 19 0 38 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) Some(JobId(1009)) Downstairs last five completed: 1009 1008 1007 1006 1005 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:08.826161015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:08.82633334Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1811,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 281 281 281 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 168 168 168 282 282 282 F F F 7 140-159 168 168 168 282 282 282 T T F <--- 8 160-179 164 164 164 276 276 276 F F F 9 180-199 167 167 167 281 281 281 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 157 157 157 259 259 259 T T F <--- 13 260-279 160 160 160 264 264 264 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 168 168 166 283 283 279 F F F <--- 16 320-339 166 166 166 280 280 280 T T F <--- 17 340-359 157 157 157 259 259 259 T T F <--- 18 360-379 167 167 167 281 281 281 T T F <--- 19 380-399 167 167 167 281 281 281 T T F <--- 20 400-419 166 166 166 279 279 279 T T F <--- 21 420-439 166 166 166 279 279 279 T T F <--- 22 440-459 160 160 160 264 264 264 F F F 23 460-479 165 165 165 277 277 277 T T F <--- 24 480-499 162 162 162 269 269 269 T T F <--- 25 500-519 167 167 167 281 281 281 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 168 168 168 282 282 282 T T F <--- 29 580-599 168 168 168 282 282 282 F F F Max gen: 168, Max flush: 283 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 281 281 281 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 168 168 168 282 282 282 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 167 167 167 281 281 281 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 169 169 169 284 284 284 F F F 13 260-279 160 160 160 264 264 264 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 168 168 168 283 283 283 F F F 16 320-339 169 169 169 284 284 284 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 169 169 169 284 284 284 F F F 19 380-399 169 169 169 284 284 284 F F F 20 400-419 169 169 169 284 284 284 F F F 21 420-439 169 169 169 284 284 284 F F F 22 440-459 160 160 160 264 264 264 F F F 23 460-479 169 169 169 284 284 284 F F F 24 480-499 169 169 169 284 284 284 F F F 25 500-519 167 167 167 281 281 281 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 169 169 169 284 284 284 F F F 29 580-599 168 168 168 282 282 282 F F F Max gen: 169, Max flush: 284 Begin loop 85 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.084866514Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.092978877Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093260572Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.09328594Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093294779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093306338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Crucible bdd309da-fe13-4887-960f-98d8cc59d416 has session id: f457abcf-fee5-4c58-9fe7-25205b94b58b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093623101Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Upstairs opts: Upstairs UUID: bdd309da-fe13-4887-960f-98d8cc59d416, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093651348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Crucible stats registered with UUID: bdd309da-fe13-4887-960f-98d8cc59d416","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093662427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:13.093671586Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093794096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093810864Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093820713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.093999578Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094016126Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"bdd309da-fe13-4887-960f-98d8cc59d416 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094048154Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094057473Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094066962Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094076531Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.09408581Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094095789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094107458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"ds_connection connected from Ok(127.0.0.1:49077)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094156484Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"ds_connection connected from Ok(127.0.0.1:64438)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094181342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"ds_connection connected from Ok(127.0.0.1:55667)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094192251Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"io task","client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094932607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094979283Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.094996971Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095256448Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095298475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[0]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.09535242Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[0]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095363839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095374758Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[1]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095385247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[1]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095395506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095408155Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[2]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095418574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[2]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095428873Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095439592Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095448932Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095458431Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Next flush: 285","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.09546772Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095476919Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"":"downstairs","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095491578Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095500237Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"0","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095509756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"1","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095519225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"client":"2","":"downstairs","session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095528925Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095537884Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"bdd309da-fe13-4887-960f-98d8cc59d416 is now active with session: f457abcf-fee5-4c58-9fe7-25205b94b58b","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095547733Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095557642Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"session_id":"f457abcf-fee5-4c58-9fe7-25205b94b58b"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095566441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Activated sub_volume bdd309da-fe13-4887-960f-98d8cc59d416","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095574161Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.09558198Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.095934149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} 01/30 Write block 427 len 16384 data: 10 14 14 15 02/30 Write block 262 len 24576 data: 15 14 16 20 20 16 03/30 Write block 312 len 32768 data: 14 13 14 14 17 16 15 18 04/30 Write block 575 len 8192 data: 9 11 05/30 Read block 37 len 20480 06/30 Write block 458 len 24576 data: 14 12 13 14 15 18 07/30 Read block 142 len 4096 08/30 Read block 142 len 28672 09/30 Write block 316 len 12288 data: 18 17 16 10/30 Flush 11/30 Write block 331 len 20480 data: 11 9 11 12 11 12/30 Flush 13/30 Flush 14/30 Write block 316 len 24576 data: 19 18 17 19 17 14 15/30 Read block 404 len 24576 16/30 Read block 520 len 16384 17/30 Read block 163 len 4096 18/30 Write block 479 len 8192 data: 14 13 19/30 Read block 173 len 28672 20/30 Read block 375 len 16384 21/30 Read block 163 len 8192 22/30 Read block 89 len 24576 23/30 Read block 472 len 32768 24/30 Read block 93 len 28672 25/30 Write block 332 len 16384 data: 10 12 13 12 26/30 Write block 472 len 12288 data: 10 11 16 27/30 Read block 53 len 4096 28/30 Write block 435 len 28672 data: 13 13 15 17 15 16 18 29/30 Write block 414 len 28672 data: 14 12 17 15 17 15 16 30/30 Write block 374 len 24576 data: 16 16 16 17 15 15 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 29 3 35 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:13.103759808Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:13.10396478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1819,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 281 281 281 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 168 168 168 282 282 282 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 167 167 167 281 281 281 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 169 169 169 284 284 284 F F F 13 260-279 170 160 170 285 264 285 F F F <--- 14 280-299 165 165 165 277 277 277 F F F 15 300-319 170 168 170 285 283 285 T F T <--- 16 320-339 170 169 170 286 284 286 T F T <--- 17 340-359 169 169 169 284 284 284 F F F 18 360-379 169 169 169 284 284 284 T F T <--- 19 380-399 169 169 169 284 284 284 F F F 20 400-419 169 169 169 284 284 284 T F T <--- 21 420-439 170 169 170 285 284 285 T T T <--- 22 440-459 170 160 170 285 264 285 T F T <--- 23 460-479 170 169 170 285 284 285 T F T <--- 24 480-499 169 169 169 284 284 284 T F T <--- 25 500-519 167 167 167 281 281 281 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 169 170 285 284 285 F F F <--- 29 580-599 168 168 168 282 282 282 F F F Max gen: 170, Max flush: 286 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 281 281 281 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 165 165 277 277 277 F F F 6 120-139 168 168 168 282 282 282 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 164 164 164 276 276 276 F F F 9 180-199 167 167 167 281 281 281 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 169 169 169 284 284 284 F F F 13 260-279 170 170 170 285 285 285 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 171 171 171 287 287 287 F F F 16 320-339 171 171 171 287 287 287 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 171 171 171 287 287 287 F F F 19 380-399 169 169 169 284 284 284 F F F 20 400-419 171 171 171 287 287 287 F F F 21 420-439 171 171 171 287 287 287 F F F 22 440-459 171 171 171 287 287 287 F F F 23 460-479 171 171 171 287 287 287 F F F 24 480-499 171 171 171 287 287 287 F F F 25 500-519 167 167 167 281 281 281 F F F 26 520-539 159 159 159 263 263 263 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 170 170 285 285 285 F F F 29 580-599 168 168 168 282 282 282 F F F Max gen: 171, Max flush: 287 Begin loop 86 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.354508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:17.36173599Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.361793225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.368792235Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369044753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369069091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.36907756Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369089089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Crucible 7010d2ae-85ab-4546-aa85-810776c76639 has session id: a5e87c28-9a80-4adc-b1b9-0eaf42d63601","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369445438Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Upstairs opts: Upstairs UUID: 7010d2ae-85ab-4546-aa85-810776c76639, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369483695Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Crucible stats registered with UUID: 7010d2ae-85ab-4546-aa85-810776c76639","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369496824Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:17.369505993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.36965012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369680668Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369691597Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369837084Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369852123Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"7010d2ae-85ab-4546-aa85-810776c76639 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.36988344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369892739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369902338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369911938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369921297Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369931096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369940855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"ds_connection connected from Ok(127.0.0.1:60640)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369968773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"ds_connection connected from Ok(127.0.0.1:63874)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.369991131Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"ds_connection connected from Ok(127.0.0.1:65426)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.370065874Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"io task","client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.370695329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.370758264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.370785532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.370997483Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[0]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371088565Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[0]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371104304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371115233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[1]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371125682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[1]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371135961Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.37114682Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[2]R flush_numbers[0..12]: [281, 274, 282, 279, 264, 277, 282, 284, 276, 281, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371157579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[2]R generation[0..12]: [167, 163, 168, 166, 160, 165, 168, 169, 164, 167, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371167908Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371180187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371189486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371198975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Next flush: 288","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371208115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371217234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"":"downstairs","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371226983Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371235652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"0","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371245371Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"1","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371254911Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"client":"2","":"downstairs","session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.37126474Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371273809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"7010d2ae-85ab-4546-aa85-810776c76639 is now active with session: a5e87c28-9a80-4adc-b1b9-0eaf42d63601","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371283528Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371292047Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"session_id":"a5e87c28-9a80-4adc-b1b9-0eaf42d63601"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371301017Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Activated sub_volume 7010d2ae-85ab-4546-aa85-810776c76639","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371308816Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371316595Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.371664885Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} 01/30 Write block 113 len 36864 data: 12 11 11 11 9 11 11 12 10 02/30 Read block 566 len 36864 03/30 Read block 545 len 4096 04/30 Flush 05/30 Write block 6 len 40960 data: 10 13 14 14 11 13 12 12 12 15 06/30 Write block 596 len 8192 data: 11 9 07/30 Read block 515 len 24576 08/30 Read block 378 len 4096 09/30 Read block 378 len 28672 10/30 Write block 371 len 12288 data: 14 13 13 11/30 Read block 70 len 20480 12/30 Read block 330 len 32768 13/30 Flush 14/30 Write block 453 len 4096 data: 15 15/30 Write block 429 len 28672 data: 15 16 14 14 12 13 14 16/30 Write block 380 len 24576 data: 16 18 17 20 16 16 17/30 Read block 240 len 28672 18/30 Write block 427 len 12288 data: 11 15 16 19/30 Write block 168 len 12288 data: 7 8 10 20/30 Write block 329 len 24576 data: 13 13 12 11 13 14 21/30 Write block 191 len 24576 data: 14 15 12 11 13 12 22/30 Flush 23/30 Flush 24/30 Write block 254 len 32768 data: 11 11 11 11 14 14 15 19 25/30 Read block 196 len 32768 26/30 Write block 448 len 40960 data: 19 17 18 17 14 16 13 14 13 10 27/30 Read block 338 len 20480 28/30 Read block 88 len 8192 29/30 Write block 519 len 28672 data: 19 18 17 16 14 16 12 30/30 Read block 174 len 24576 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:17.381034799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:17.381216333Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1827,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 172 172 281 289 289 F F F <--- 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 165 172 172 277 288 288 T F F <--- 6 120-139 168 172 172 282 288 288 T F F <--- 7 140-159 169 169 169 284 284 284 F F F 8 160-179 164 172 172 276 290 290 F F F <--- 9 180-199 167 172 172 281 290 290 F F F <--- 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 169 169 169 284 284 284 F T T <--- 13 260-279 170 170 170 285 285 285 F T T <--- 14 280-299 165 165 165 277 277 277 F F F 15 300-319 171 171 171 287 287 287 F F F 16 320-339 171 172 172 287 290 290 F F F <--- 17 340-359 169 169 169 284 284 284 F F F 18 360-379 171 172 172 287 289 289 F F F <--- 19 380-399 169 172 172 284 290 290 F F F <--- 20 400-419 171 171 171 287 287 287 F F F 21 420-439 171 172 172 287 290 290 F F F <--- 22 440-459 171 172 172 287 290 290 F T T <--- 23 460-479 171 171 171 287 287 287 F F F 24 480-499 171 171 171 287 287 287 F F F 25 500-519 167 167 167 281 281 281 F T T <--- 26 520-539 159 159 159 263 263 263 F T T <--- 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 170 170 285 285 285 F F F 29 580-599 168 172 172 282 289 289 F F F <--- Max gen: 172, Max flush: 290 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 289 289 289 F F F 1 020-039 163 163 163 274 274 274 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 160 160 264 264 264 F F F 5 100-119 172 172 172 288 288 288 F F F 6 120-139 172 172 172 288 288 288 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 167 167 167 281 281 281 F F F 11 220-239 158 158 158 261 261 261 F F F 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 165 165 165 277 277 277 F F F 15 300-319 171 171 171 287 287 287 F F F 16 320-339 172 172 172 290 290 290 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 172 172 172 289 289 289 F F F 19 380-399 172 172 172 290 290 290 F F F 20 400-419 171 171 171 287 287 287 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 171 171 171 287 287 287 F F F 24 480-499 171 171 171 287 287 287 F F F 25 500-519 173 173 173 291 291 291 F F F 26 520-539 173 173 173 291 291 291 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 170 170 285 285 285 F F F 29 580-599 172 172 172 289 289 289 F F F Max gen: 173, Max flush: 291 Begin loop 87 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.648121576Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:21.655381694Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.655449568Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.662524821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.662800787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.662826935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.662835274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.662846643Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Crucible 9868e5c4-e58b-4a87-9952-1b28cf761861 has session id: 5376bbdb-c657-46e1-bf7e-d5045295fa54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663249038Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Upstairs opts: Upstairs UUID: 9868e5c4-e58b-4a87-9952-1b28cf761861, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663271506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Crucible stats registered with UUID: 9868e5c4-e58b-4a87-9952-1b28cf761861","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663281605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:21.663290655Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663444341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663475759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663486398Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663611757Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663636794Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"9868e5c4-e58b-4a87-9952-1b28cf761861 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663654083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663662822Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663672331Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663681931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.66369136Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663701389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663711328Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:53616)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663780382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:43080)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.663834687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:57735)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.66391721Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"io task","client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664473972Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664533846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664547045Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664803443Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664849549Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[0]R flush_numbers[0..12]: [289, 274, 282, 279, 264, 288, 288, 284, 290, 290, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664890425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[0]R generation[0..12]: [172, 163, 168, 166, 160, 172, 172, 169, 172, 172, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664901314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664912103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[1]R flush_numbers[0..12]: [289, 274, 282, 279, 264, 288, 288, 284, 290, 290, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664922602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[1]R generation[0..12]: [172, 163, 168, 166, 160, 172, 172, 169, 172, 172, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664932932Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664943491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[2]R flush_numbers[0..12]: [289, 274, 282, 279, 264, 288, 288, 284, 290, 290, 281, 261]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.66495411Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[2]R generation[0..12]: [172, 163, 168, 166, 160, 172, 172, 169, 172, 172, 167, 158]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664964559Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664975248Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664984497Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.664994066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Next flush: 292","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665004705Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665013975Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"":"downstairs","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665023714Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665032393Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"0","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665042192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"1","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665051841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"client":"2","":"downstairs","session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.66506167Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.66507071Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"9868e5c4-e58b-4a87-9952-1b28cf761861 is now active with session: 5376bbdb-c657-46e1-bf7e-d5045295fa54","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665080409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665088898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"session_id":"5376bbdb-c657-46e1-bf7e-d5045295fa54"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665097737Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Activated sub_volume 9868e5c4-e58b-4a87-9952-1b28cf761861","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665105447Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665113486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.665471075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} 01/30 Write block 92 len 8192 data: 15 12 02/30 Read block 402 len 16384 03/30 Write block 493 len 40960 data: 9 7 8 8 9 12 13 8 8 10 04/30 Read block 134 len 32768 05/30 Write block 101 len 36864 data: 12 12 14 15 16 12 13 11 8 06/30 Read block 558 len 16384 07/30 Write block 480 len 16384 data: 14 15 17 16 08/30 Write block 317 len 16384 data: 19 18 20 18 09/30 Write block 410 len 24576 data: 13 10 10 10 15 13 10/30 Read block 417 len 4096 11/30 Write block 473 len 28672 data: 12 17 18 14 13 17 15 12/30 Read block 470 len 8192 13/30 Read block 518 len 16384 14/30 Write block 214 len 32768 data: 10 10 10 12 13 14 12 10 15/30 Read block 282 len 4096 16/30 Read block 213 len 4096 17/30 Read block 417 len 8192 18/30 Read block 304 len 32768 19/30 Read block 155 len 4096 20/30 Write block 298 len 16384 data: 15 19 15 14 21/30 Flush 22/30 Write block 212 len 12288 data: 14 13 11 23/30 Read block 249 len 20480 24/30 Write block 302 len 16384 data: 18 19 19 17 25/30 Read block 485 len 20480 26/30 Flush 27/30 Write block 464 len 4096 data: 18 28/30 Read block 428 len 4096 29/30 Write block 24 len 12288 data: 11 11 13 30/30 Write block 18 len 20480 data: 13 14 12 8 8 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 2 2 28 Done 6 28 28 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:21.673732675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:21.673896591Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1835,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 289 289 289 F T T <--- 1 020-039 163 163 163 274 274 274 F T T <--- 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 160 174 174 264 292 292 T F F <--- 5 100-119 172 174 174 288 292 292 T F F <--- 6 120-139 172 172 172 288 288 288 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 167 174 174 281 293 293 F F F <--- 11 220-239 158 174 174 261 292 292 F F F <--- 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 165 174 174 277 292 292 F F F <--- 15 300-319 171 174 174 287 293 293 F F F <--- 16 320-339 172 174 174 290 292 292 F F F <--- 17 340-359 169 169 169 284 284 284 F F F 18 360-379 172 172 172 289 289 289 F F F 19 380-399 172 172 172 290 290 290 F F F 20 400-419 171 174 174 287 292 292 F F F <--- 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 171 174 174 287 292 292 F T T <--- 24 480-499 171 174 174 287 292 292 T F F <--- 25 500-519 173 174 174 291 292 292 T F F <--- 26 520-539 173 173 173 291 291 291 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 170 170 285 285 285 F F F 29 580-599 172 172 172 289 289 289 F F F Max gen: 174, Max flush: 293 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 294 294 294 F F F 1 020-039 175 175 175 294 294 294 F F F 2 040-059 168 168 168 282 282 282 F F F 3 060-079 166 166 166 279 279 279 F F F 4 080-099 174 174 174 292 292 292 F F F 5 100-119 174 174 174 292 292 292 F F F 6 120-139 172 172 172 288 288 288 F F F 7 140-159 169 169 169 284 284 284 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 174 174 174 293 293 293 F F F 11 220-239 174 174 174 292 292 292 F F F 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 174 174 174 293 293 293 F F F 16 320-339 174 174 174 292 292 292 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 172 172 172 289 289 289 F F F 19 380-399 172 172 172 290 290 290 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 175 175 175 294 294 294 F F F 24 480-499 174 174 174 292 292 292 F F F 25 500-519 174 174 174 292 292 292 F F F 26 520-539 173 173 173 291 291 291 F F F 27 540-559 166 166 166 279 279 279 F F F 28 560-579 170 170 170 285 285 285 F F F 29 580-599 172 172 172 289 289 289 F F F Max gen: 175, Max flush: 294 Begin loop 88 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.934622142Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:25.94176499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.941820805Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.948663149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.948920247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.948945254Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.948953364Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.948964753Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Crucible 5249245c-e756-418a-a186-f40798139c56 has session id: b882ac1e-e2d8-4d6e-8a62-10885055ea60","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949319632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Upstairs opts: Upstairs UUID: 5249245c-e756-418a-a186-f40798139c56, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949349419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Crucible stats registered with UUID: 5249245c-e756-418a-a186-f40798139c56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949360138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:25.949369117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949486627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949519374Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949530243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949654523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949678651Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"5249245c-e756-418a-a186-f40798139c56 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949692789Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949701669Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949711228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949720637Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949729856Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949739645Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"ds_connection connected from Ok(127.0.0.1:59651)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949782341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949792561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"ds_connection connected from Ok(127.0.0.1:55844)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.94980273Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"ds_connection connected from Ok(127.0.0.1:60029)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.949861135Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"io task","client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950481361Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950509508Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950520567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950817751Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950842429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[0]R flush_numbers[0..12]: [294, 294, 282, 279, 292, 292, 288, 284, 290, 290, 293, 292]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950904604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[0]R generation[0..12]: [175, 175, 168, 166, 174, 174, 172, 169, 172, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950917713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950928562Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[1]R flush_numbers[0..12]: [294, 294, 282, 279, 292, 292, 288, 284, 290, 290, 293, 292]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950938951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[1]R generation[0..12]: [175, 175, 168, 166, 174, 174, 172, 169, 172, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.95094941Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950960349Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[2]R flush_numbers[0..12]: [294, 294, 282, 279, 292, 292, 288, 284, 290, 290, 293, 292]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950970688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[2]R generation[0..12]: [175, 175, 168, 166, 174, 174, 172, 169, 172, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950980907Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.950991546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951000695Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951010454Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Next flush: 295","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951019624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951028763Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"":"downstairs","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951038342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951047051Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"0","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.9510579Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"1","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951067619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"client":"2","":"downstairs","session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951077149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951086108Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"5249245c-e756-418a-a186-f40798139c56 is now active with session: b882ac1e-e2d8-4d6e-8a62-10885055ea60","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951095747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951104186Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"session_id":"b882ac1e-e2d8-4d6e-8a62-10885055ea60"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951113036Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Activated sub_volume 5249245c-e756-418a-a186-f40798139c56","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951120745Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951134194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.951418809Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} 01/30 Write block 150 len 8192 data: 12 13 02/30 Flush 03/30 Write block 566 len 24576 data: 10 8 8 12 11 12 04/30 Read block 417 len 4096 05/30 Write block 20 len 36864 data: 13 9 9 8 12 12 14 11 12 06/30 Read block 165 len 24576 07/30 Read block 403 len 16384 08/30 Read block 372 len 12288 09/30 Read block 270 len 28672 10/30 Flush 11/30 Write block 61 len 12288 data: 15 10 13 12/30 Read block 486 len 24576 13/30 Write block 233 len 4096 data: 16 14/30 Flush 15/30 Write block 365 len 40960 data: 13 11 14 18 18 13 15 14 14 17 16/30 Write block 29 len 28672 data: 8 8 10 10 12 12 14 17/30 Read block 477 len 12288 18/30 Write block 49 len 40960 data: 18 19 18 16 15 18 15 11 10 9 19/30 Write block 554 len 32768 data: 12 15 13 12 12 12 10 12 20/30 Flush 21/30 Flush 22/30 Write block 584 len 4096 data: 12 23/30 Write block 504 len 32768 data: 13 14 17 17 17 14 13 13 24/30 Write block 375 len 20480 data: 17 17 18 16 16 25/30 Read block 24 len 36864 26/30 Read block 569 len 12288 27/30 Read block 74 len 4096 28/30 Read block 449 len 20480 29/30 Write block 337 len 24576 data: 17 20 18 16 16 15 30/30 Read block 515 len 12288 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 6 Done Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 1 28 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:25.960092843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:25.960270768Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1843,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 294 294 294 F F F 1 020-039 175 176 176 294 298 298 F F F <--- 2 040-059 168 176 176 282 298 298 F F F <--- 3 060-079 166 176 176 279 297 297 F F F <--- 4 080-099 174 174 174 292 292 292 F F F 5 100-119 174 174 174 292 292 292 F F F 6 120-139 172 172 172 288 288 288 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 174 174 174 293 293 293 F F F 11 220-239 174 176 176 292 297 297 F F F <--- 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 174 174 174 293 293 293 F F F 16 320-339 174 174 174 292 292 292 F T T <--- 17 340-359 169 169 169 284 284 284 F T T <--- 18 360-379 172 176 176 289 298 298 F T T <--- 19 380-399 172 172 172 290 290 290 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 175 175 175 294 294 294 F F F 24 480-499 174 174 174 292 292 292 F F F 25 500-519 174 174 174 292 292 292 F T T <--- 26 520-539 173 173 173 291 291 291 F F F 27 540-559 166 176 176 279 298 298 F F F <--- 28 560-579 170 176 176 285 298 298 T F F <--- 29 580-599 172 172 172 289 289 289 F T T <--- Max gen: 176, Max flush: 298 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 294 294 294 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 176 176 176 298 298 298 F F F 3 060-079 176 176 176 297 297 297 F F F 4 080-099 174 174 174 292 292 292 F F F 5 100-119 174 174 174 292 292 292 F F F 6 120-139 172 172 172 288 288 288 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 174 174 174 293 293 293 F F F 11 220-239 176 176 176 297 297 297 F F F 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 174 174 174 293 293 293 F F F 16 320-339 177 177 177 299 299 299 F F F 17 340-359 177 177 177 299 299 299 F F F 18 360-379 177 177 177 299 299 299 F F F 19 380-399 172 172 172 290 290 290 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 175 175 175 294 294 294 F F F 24 480-499 174 174 174 292 292 292 F F F 25 500-519 177 177 177 299 299 299 F F F 26 520-539 173 173 173 291 291 291 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 176 176 176 298 298 298 F F F 29 580-599 177 177 177 299 299 299 F F F Max gen: 177, Max flush: 299 Begin loop 89 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.218608868Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227073891Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227372305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227398152Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227406512Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227418071Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Crucible 3f221750-45b2-4195-8265-a8ff93e6d3de has session id: 918ec17c-09bc-478f-8913-d9756d2aea48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.22776641Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Upstairs opts: Upstairs UUID: 3f221750-45b2-4195-8265-a8ff93e6d3de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227794308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Crucible stats registered with UUID: 3f221750-45b2-4195-8265-a8ff93e6d3de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227805117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:30.227814196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227949634Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227978532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.227989281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228120249Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228144117Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"3f221750-45b2-4195-8265-a8ff93e6d3de active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228168235Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228177324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228186994Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228196553Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228205832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.2282238Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228235979Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"ds_connection connected from Ok(127.0.0.1:60147)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228262577Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"ds_connection connected from Ok(127.0.0.1:48985)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228291924Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"ds_connection connected from Ok(127.0.0.1:64478)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.228394185Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"io task","client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229016601Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229039739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229050708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229354352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.22937787Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[0]R flush_numbers[0..12]: [294, 298, 298, 297, 292, 292, 288, 295, 290, 290, 293, 297]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229420726Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[0]R generation[0..12]: [175, 176, 176, 176, 174, 174, 172, 176, 172, 172, 174, 176]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229432585Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229443524Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[1]R flush_numbers[0..12]: [294, 298, 298, 297, 292, 292, 288, 295, 290, 290, 293, 297]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229454233Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[1]R generation[0..12]: [175, 176, 176, 176, 174, 174, 172, 176, 172, 172, 174, 176]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229464662Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229475381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[2]R flush_numbers[0..12]: [294, 298, 298, 297, 292, 292, 288, 295, 290, 290, 293, 297]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.22948796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[2]R generation[0..12]: [175, 176, 176, 176, 174, 174, 172, 176, 172, 172, 174, 176]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229498519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229509218Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229518478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229528087Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Next flush: 300","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229537316Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229546415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"":"downstairs","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229555904Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229564574Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"0","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229574113Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"1","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229583462Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"client":"2","":"downstairs","session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229593101Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.22960193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"3f221750-45b2-4195-8265-a8ff93e6d3de is now active with session: 918ec17c-09bc-478f-8913-d9756d2aea48","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229611679Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229620299Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"session_id":"918ec17c-09bc-478f-8913-d9756d2aea48"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229630538Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Activated sub_volume 3f221750-45b2-4195-8265-a8ff93e6d3de","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229638387Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229646226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.229980627Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} 01/30 Write block 323 len 36864 data: 18 17 17 16 15 16 14 14 13 02/30 Write block 362 len 8192 data: 16 15 03/30 Read block 4 len 28672 04/30 Write block 71 len 36864 data: 13 11 8 11 12 10 11 11 11 05/30 Read block 264 len 4096 06/30 Write block 328 len 40960 data: 17 15 15 14 12 14 15 13 13 18 07/30 Read block 200 len 28672 08/30 Read block 157 len 36864 09/30 Write block 133 len 16384 data: 15 15 16 17 10/30 Flush 11/30 Write block 58 len 28672 data: 10 11 11 16 11 14 12 12/30 Write block 70 len 12288 data: 13 14 12 13/30 Read block 186 len 40960 14/30 Read block 339 len 24576 15/30 Write block 101 len 8192 data: 13 13 16/30 Read block 238 len 36864 17/30 Write block 397 len 12288 data: 18 20 20 18/30 Write block 8 len 16384 data: 15 15 12 14 19/30 Write block 224 len 40960 data: 6 7 11 12 15 15 16 18 18 17 20/30 Read block 332 len 28672 21/30 Write block 496 len 8192 data: 9 10 22/30 Flush 23/30 Flush 24/30 Write block 85 len 36864 data: 14 16 14 20 20 17 14 16 13 25/30 Flush 26/30 Write block 82 len 12288 data: 10 11 12 27/30 Write block 126 len 8192 data: 12 13 28/30 Read block 26 len 20480 29/30 Write block 594 len 8192 data: 11 11 30/30 Read block 130 len 16384 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 23 0 23 Done 30 7 30 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:30.239045318Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:30.239220802Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1851,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 175 178 301 294 301 F F F <--- 1 020-039 176 176 176 298 298 298 F F F 2 040-059 178 176 178 301 298 301 F F F <--- 3 060-079 178 176 178 301 297 301 F T F <--- 4 080-099 178 174 178 303 292 303 T F T <--- 5 100-119 178 174 178 301 292 301 F F F <--- 6 120-139 178 172 178 300 288 300 T F T <--- 7 140-159 176 176 176 295 295 295 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 174 174 174 293 293 293 F F F 11 220-239 178 176 178 301 297 301 F F F <--- 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 174 174 174 293 293 293 F F F 16 320-339 178 177 178 300 299 300 F T F <--- 17 340-359 177 177 177 299 299 299 F F F 18 360-379 178 177 178 300 299 300 F T F <--- 19 380-399 178 172 178 301 290 301 F F F <--- 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 175 175 175 294 294 294 F F F 24 480-499 178 174 178 301 292 301 F F F <--- 25 500-519 177 177 177 299 299 299 F F F 26 520-539 173 173 173 291 291 291 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 176 176 176 298 298 298 F F F 29 580-599 177 177 177 299 299 299 T F T <--- Max gen: 178, Max flush: 303 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 301 301 301 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 178 178 178 301 301 301 F F F 3 060-079 178 178 178 301 301 301 F F F 4 080-099 179 179 179 304 304 304 F F F 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 172 172 172 290 290 290 F F F 9 180-199 172 172 172 290 290 290 F F F 10 200-219 174 174 174 293 293 293 F F F 11 220-239 178 178 178 301 301 301 F F F 12 240-259 173 173 173 291 291 291 F F F 13 260-279 173 173 173 291 291 291 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 174 174 174 293 293 293 F F F 16 320-339 178 178 178 300 300 300 F F F 17 340-359 177 177 177 299 299 299 F F F 18 360-379 178 178 178 300 300 300 F F F 19 380-399 178 178 178 301 301 301 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 173 173 173 291 291 291 F F F 23 460-479 175 175 175 294 294 294 F F F 24 480-499 178 178 178 301 301 301 F F F 25 500-519 177 177 177 299 299 299 F F F 26 520-539 173 173 173 291 291 291 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 176 176 176 298 298 298 F F F 29 580-599 179 179 179 304 304 304 F F F Max gen: 179, Max flush: 304 Begin loop 90 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.490230281Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.498422817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.498703013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.498729171Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.49873743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.498749019Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Crucible 1b4b5ba8-c8fa-445d-abab-2ca27113bb32 has session id: e7a9441b-87b7-42ba-a91b-dbe22fd4c053","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499088149Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Upstairs opts: Upstairs UUID: 1b4b5ba8-c8fa-445d-abab-2ca27113bb32, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499118197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Crucible stats registered with UUID: 1b4b5ba8-c8fa-445d-abab-2ca27113bb32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499128646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:34.499137785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499288082Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499321609Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499333348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499463247Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499478205Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"1b4b5ba8-c8fa-445d-abab-2ca27113bb32 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499516042Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499526421Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.49953707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499547779Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499558228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499568948Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499579967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"ds_connection connected from Ok(127.0.0.1:63112)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499672109Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"ds_connection connected from Ok(127.0.0.1:43310)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499697986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"ds_connection connected from Ok(127.0.0.1:64191)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.499756861Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"io task","client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500274916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500292375Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500303204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500606067Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500615526Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[0]R flush_numbers[0..12]: [301, 298, 301, 301, 304, 301, 304, 295, 290, 290, 293, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500649133Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[0]R generation[0..12]: [178, 176, 178, 178, 179, 178, 179, 176, 172, 172, 174, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500660162Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500671002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[1]R flush_numbers[0..12]: [301, 298, 301, 301, 304, 301, 304, 295, 290, 290, 293, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500681381Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[1]R generation[0..12]: [178, 176, 178, 178, 179, 178, 179, 176, 172, 172, 174, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.50069165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500702699Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[2]R flush_numbers[0..12]: [301, 298, 301, 301, 304, 301, 304, 295, 290, 290, 293, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500713208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[2]R generation[0..12]: [178, 176, 178, 178, 179, 178, 179, 176, 172, 172, 174, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500723567Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500734286Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500743635Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500754534Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Next flush: 305","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500763713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500772803Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"":"downstairs","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500782532Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500791151Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"0","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.50080083Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"1","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500810189Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"client":"2","":"downstairs","session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500819749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500828618Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"1b4b5ba8-c8fa-445d-abab-2ca27113bb32 is now active with session: e7a9441b-87b7-42ba-a91b-dbe22fd4c053","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500838207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500846606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"session_id":"e7a9441b-87b7-42ba-a91b-dbe22fd4c053"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500855615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Activated sub_volume 1b4b5ba8-c8fa-445d-abab-2ca27113bb32","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500863405Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.500871234Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.501234542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} 01/30 Write block 303 len 36864 data: 20 20 18 18 18 16 15 16 15 02/30 Flush 03/30 Write block 58 len 20480 data: 11 12 12 17 12 04/30 Read block 525 len 16384 05/30 Read block 434 len 36864 06/30 Read block 352 len 24576 07/30 Read block 523 len 16384 08/30 Read block 110 len 4096 09/30 Write block 512 len 4096 data: 16 10/30 Write block 488 len 20480 data: 13 11 10 8 8 11/30 Read block 79 len 20480 12/30 Flush 13/30 Write block 473 len 24576 data: 13 18 19 15 14 18 14/30 Read block 416 len 24576 15/30 Read block 387 len 12288 16/30 Write block 451 len 20480 data: 18 15 17 14 15 17/30 Write block 387 len 32768 data: 22 21 21 19 21 27 21 25 18/30 Write block 301 len 16384 data: 15 19 21 21 19/30 Write block 254 len 40960 data: 12 12 12 12 15 15 16 20 16 15 20/30 Read block 317 len 24576 21/30 Write block 305 len 8192 data: 19 19 22/30 Write block 160 len 36864 data: 14 11 11 9 8 9 7 6 8 23/30 Read block 362 len 32768 24/30 Read block 289 len 8192 25/30 Flush 26/30 Write block 565 len 28672 data: 9 11 9 9 13 12 13 27/30 Write block 72 len 16384 data: 13 9 12 13 28/30 Write block 535 len 8192 data: 14 14 29/30 Write block 85 len 36864 data: 15 17 15 21 21 18 15 17 14 30/30 Write block 192 len 40960 data: 16 13 12 14 13 12 10 8 7 7 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 5 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 25 2 29 Done 26 3 26 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1001)) Some(JobId(1024)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:34.509723073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:34.509908867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1859,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 301 301 301 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 178 180 306 301 306 F T F <--- 3 060-079 180 178 180 306 301 306 T T T <--- 4 080-099 179 179 179 304 304 304 T F T <--- 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 180 172 180 307 290 307 F F F <--- 9 180-199 172 172 172 290 290 290 T F T <--- 10 200-219 174 174 174 293 293 293 T F T <--- 11 220-239 178 178 178 301 301 301 F F F 12 240-259 180 173 180 307 291 307 F F F <--- 13 260-279 180 173 180 307 291 307 F F F <--- 14 280-299 174 174 174 292 292 292 F F F 15 300-319 180 180 180 307 305 307 F F F <--- 16 320-339 178 178 178 300 300 300 F F F 17 340-359 177 177 177 299 299 299 F F F 18 360-379 178 178 178 300 300 300 F F F 19 380-399 180 178 180 307 301 307 F F F <--- 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 180 173 180 307 291 307 F F F <--- 23 460-479 180 175 180 307 294 307 F F F <--- 24 480-499 180 178 180 306 301 306 F F F <--- 25 500-519 180 177 180 306 299 306 F F F <--- 26 520-539 173 173 173 291 291 291 T F T <--- 27 540-559 176 176 176 298 298 298 F F F 28 560-579 176 176 176 298 298 298 T F T <--- 29 580-599 179 179 179 304 304 304 F F F Max gen: 180, Max flush: 307 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 301 301 301 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 180 180 306 306 306 F F F 3 060-079 181 181 181 308 308 308 F F F 4 080-099 181 181 181 308 308 308 F F F 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 180 180 180 307 307 307 F F F 9 180-199 181 181 181 308 308 308 F F F 10 200-219 181 181 181 308 308 308 F F F 11 220-239 178 178 178 301 301 301 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 180 180 180 307 307 307 F F F 14 280-299 174 174 174 292 292 292 F F F 15 300-319 180 180 180 307 307 307 F F F 16 320-339 178 178 178 300 300 300 F F F 17 340-359 177 177 177 299 299 299 F F F 18 360-379 178 178 178 300 300 300 F F F 19 380-399 180 180 180 307 307 307 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 F F F 22 440-459 180 180 180 307 307 307 F F F 23 460-479 180 180 180 307 307 307 F F F 24 480-499 180 180 180 306 306 306 F F F 25 500-519 180 180 180 306 306 306 F F F 26 520-539 181 181 181 308 308 308 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 181 181 181 308 308 308 F F F 29 580-599 179 179 179 304 304 304 F F F Max gen: 181, Max flush: 308 Begin loop 91 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.771640812Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.779847817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780122533Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780146051Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.78015476Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780166279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Crucible 02d76e01-ed9e-44ed-9955-29411c11a801 has session id: 975b08bf-21d8-4d13-8732-cf490c81091e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780520488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Upstairs opts: Upstairs UUID: 02d76e01-ed9e-44ed-9955-29411c11a801, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780552056Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Crucible stats registered with UUID: 02d76e01-ed9e-44ed-9955-29411c11a801","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780564124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:38.780574344Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780742829Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780773006Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780783685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780881687Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780905545Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"02d76e01-ed9e-44ed-9955-29411c11a801 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780922193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780933062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780942551Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780951841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.78096108Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780970989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.780980558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44106)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781007016Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58030)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781024664Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"ds_connection connected from Ok(127.0.0.1:33568)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781090089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"io task","client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781682937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781706045Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.781734422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782011548Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782054065Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[0]R flush_numbers[0..12]: [301, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782143967Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[0]R generation[0..12]: [178, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782158866Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782169935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[1]R flush_numbers[0..12]: [301, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782180564Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[1]R generation[0..12]: [178, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782191013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782201842Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[2]R flush_numbers[0..12]: [301, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 301]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782212211Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[2]R generation[0..12]: [178, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 178]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.7822226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782233499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782242778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782252507Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Next flush: 309","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782261717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782270896Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"":"downstairs","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782280615Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782289264Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"0","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782298833Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"1","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782309422Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"client":"2","":"downstairs","session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782319072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782327971Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"02d76e01-ed9e-44ed-9955-29411c11a801 is now active with session: 975b08bf-21d8-4d13-8732-cf490c81091e","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.78233776Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782346489Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"session_id":"975b08bf-21d8-4d13-8732-cf490c81091e"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782355338Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Activated sub_volume 02d76e01-ed9e-44ed-9955-29411c11a801","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782363168Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782370987Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.782698918Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} 01/30 Write block 503 len 28672 data: 11 14 15 18 18 18 15 02/30 Read block 508 len 32768 03/30 Read block 359 len 16384 04/30 Read block 362 len 8192 05/30 Read block 368 len 28672 06/30 Read block 110 len 36864 07/30 Read block 255 len 8192 08/30 Read block 187 len 24576 09/30 Write block 464 len 36864 data: 19 16 14 12 11 13 13 10 11 10/30 Write block 481 len 20480 data: 16 18 17 15 12 11/30 Read block 511 len 16384 12/30 Write block 346 len 20480 data: 12 10 10 12 15 13/30 Write block 13 len 20480 data: 13 13 16 13 13 14/30 Write block 594 len 16384 data: 12 12 12 10 15/30 Flush 16/30 Write block 568 len 36864 data: 10 14 13 14 12 10 10 10 12 17/30 Read block 408 len 32768 18/30 Read block 111 len 36864 19/30 Write block 281 len 32768 data: 14 12 10 11 11 11 11 11 20/30 Read block 549 len 20480 21/30 Write block 299 len 28672 data: 20 16 16 20 22 22 20 22/30 Read block 596 len 8192 23/30 Write block 434 len 28672 data: 14 15 14 16 18 16 17 24/30 Write block 316 len 20480 data: 20 20 19 21 19 25/30 Write block 223 len 4096 data: 5 26/30 Read block 239 len 4096 27/30 Write block 478 len 20480 data: 19 16 15 17 19 28/30 Read block 268 len 24576 29/30 Read block 267 len 16384 30/30 Read block 334 len 32768 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 4 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 9 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:38.791901377Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:38.792086811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1867,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 178 309 309 301 F F F <--- 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 180 180 306 306 306 F F F 3 060-079 181 181 181 308 308 308 F F F 4 080-099 181 181 181 308 308 308 F F F 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 180 180 180 307 307 307 F F F 9 180-199 181 181 181 308 308 308 F F F 10 200-219 181 181 181 308 308 308 F F F 11 220-239 178 178 178 301 301 301 T T F <--- 12 240-259 180 180 180 307 307 307 F F F 13 260-279 180 180 180 307 307 307 F F F 14 280-299 174 174 174 292 292 292 T T F <--- 15 300-319 180 180 180 307 307 307 T T F <--- 16 320-339 178 178 178 300 300 300 T T F <--- 17 340-359 182 182 177 309 309 299 F F F <--- 18 360-379 178 178 178 300 300 300 F F F 19 380-399 180 180 180 307 307 307 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 172 172 172 290 290 290 T T F <--- 22 440-459 180 180 180 307 307 307 T T F <--- 23 460-479 182 182 180 309 309 307 T T F <--- 24 480-499 182 182 180 309 309 306 T T F <--- 25 500-519 182 182 180 309 309 306 F F T <--- 26 520-539 181 181 181 308 308 308 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 181 181 181 308 308 308 T T F <--- 29 580-599 182 182 179 309 309 304 F F F <--- Max gen: 182, Max flush: 309 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 309 309 309 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 180 180 306 306 306 F F F 3 060-079 181 181 181 308 308 308 F F F 4 080-099 181 181 181 308 308 308 F F F 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 180 180 180 307 307 307 F F F 9 180-199 181 181 181 308 308 308 F F F 10 200-219 181 181 181 308 308 308 F F F 11 220-239 183 183 183 310 310 310 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 180 180 180 307 307 307 F F F 14 280-299 183 183 183 310 310 310 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 178 178 178 300 300 300 F F F 19 380-399 180 180 180 307 307 307 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 183 183 310 310 310 F F F 23 460-479 183 183 183 310 310 310 F F F 24 480-499 183 183 183 310 310 310 F F F 25 500-519 182 182 182 309 309 309 F F F 26 520-539 181 181 181 308 308 308 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 183 183 183 310 310 310 F F F 29 580-599 182 182 182 309 309 309 F F F Max gen: 183, Max flush: 310 Begin loop 92 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.050253956Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.058415315Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.058691841Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.058717339Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.058729778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.058741097Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Crucible 66784c39-8980-4b1a-bcdd-7a1359080396 has session id: 17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059069518Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Upstairs opts: Upstairs UUID: 66784c39-8980-4b1a-bcdd-7a1359080396, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059098466Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Crucible stats registered with UUID: 66784c39-8980-4b1a-bcdd-7a1359080396","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059109435Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:43.059118614Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059225015Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059249942Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059260201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059428037Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059460984Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"66784c39-8980-4b1a-bcdd-7a1359080396 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059469493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059478322Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059487742Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059497191Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.05950654Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059516519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059545607Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:36118)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059595492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:53928)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.05962312Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:45112)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.059635299Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"io task","client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060248155Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060265494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060303811Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060577887Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060587406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[0]R flush_numbers[0..12]: [309, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060600735Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[0]R generation[0..12]: [182, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060611224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060621993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[1]R flush_numbers[0..12]: [309, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060632292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[1]R generation[0..12]: [182, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060642471Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.06065304Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[2]R flush_numbers[0..12]: [309, 298, 306, 308, 308, 301, 304, 295, 307, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060665749Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[2]R generation[0..12]: [182, 176, 180, 181, 181, 178, 179, 176, 180, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060676138Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060686867Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060696146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060705846Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Next flush: 311","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060714935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060724004Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"":"downstairs","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060733543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060742092Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"0","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060751542Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"1","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060760981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"client":"2","":"downstairs","session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.06077055Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060784409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"66784c39-8980-4b1a-bcdd-7a1359080396 is now active with session: 17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060794488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060803127Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"session_id":"17d1fe8d-6f4a-418b-b523-bc9dd2f2bb1f"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060813156Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Activated sub_volume 66784c39-8980-4b1a-bcdd-7a1359080396","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060821035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.060828915Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.061145307Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} 01/30 Write block 579 len 4096 data: 6 02/30 Write block 582 len 32768 data: 8 7 13 14 17 16 16 17 03/30 Read block 422 len 24576 04/30 Write block 161 len 24576 data: 12 12 10 9 10 8 05/30 Read block 357 len 4096 06/30 Read block 236 len 4096 07/30 Write block 506 len 4096 data: 19 08/30 Read block 491 len 24576 09/30 Read block 111 len 20480 10/30 Read block 73 len 28672 11/30 Read block 72 len 4096 12/30 Read block 230 len 36864 13/30 Flush 14/30 Write block 475 len 16384 data: 20 16 15 20 15/30 Read block 86 len 32768 16/30 Read block 354 len 36864 17/30 Read block 220 len 24576 18/30 Write block 513 len 24576 data: 19 18 17 19 20 18 19/30 Write block 67 len 12288 data: 14 14 12 20/30 Read block 370 len 32768 21/30 Write block 262 len 8192 data: 17 16 22/30 Write block 574 len 20480 data: 11 11 13 9 6 23/30 Write block 90 len 28672 data: 19 16 18 15 11 11 9 24/30 Write block 596 len 16384 data: 13 11 8 4 25/30 Write block 364 len 20480 data: 13 14 12 15 19 26/30 Write block 386 len 12288 data: 18 23 22 27/30 Read block 501 len 32768 28/30 Read block 85 len 12288 29/30 Read block 133 len 36864 30/30 Read block 486 len 36864 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Sent Done false 1001 Acked Write 8 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:43.068542603Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:43.068719897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1875,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 309 309 309 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 180 180 306 306 306 F F F 3 060-079 181 181 181 308 308 308 T F T <--- 4 080-099 181 181 181 308 308 308 T F T <--- 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 184 180 184 311 307 311 F F F <--- 9 180-199 181 181 181 308 308 308 F F F 10 200-219 181 181 181 308 308 308 F F F 11 220-239 183 183 183 310 310 310 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 180 180 180 307 307 307 T F T <--- 14 280-299 183 183 183 310 310 310 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 178 178 178 300 300 300 T F T <--- 19 380-399 180 180 180 307 307 307 T F T <--- 20 400-419 174 174 174 292 292 292 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 183 183 310 310 310 F F F 23 460-479 183 183 183 310 310 310 T F T <--- 24 480-499 183 183 183 310 310 310 F F F 25 500-519 184 182 184 311 309 311 T F T <--- 26 520-539 181 181 181 308 308 308 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 184 183 184 311 310 311 T F T <--- 29 580-599 184 182 184 311 309 311 T F T <--- Max gen: 184, Max flush: 311 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 309 309 309 F F F 1 020-039 176 176 176 298 298 298 F F F 2 040-059 180 180 180 306 306 306 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 185 185 185 312 312 312 F F F 5 100-119 178 178 178 301 301 301 F F F 6 120-139 179 179 179 304 304 304 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 184 184 184 311 311 311 F F F 9 180-199 181 181 181 308 308 308 F F F 10 200-219 181 181 181 308 308 308 F F F 11 220-239 183 183 183 310 310 310 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 185 185 185 312 312 312 F F F 14 280-299 183 183 183 310 310 310 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 185 185 185 312 312 312 F F F 19 380-399 185 185 185 312 312 312 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 183 183 310 310 310 F F F 23 460-479 185 185 185 312 312 312 F F F 24 480-499 183 183 183 310 310 310 F F F 25 500-519 185 185 185 312 312 312 F F F 26 520-539 181 181 181 308 308 308 F F F 27 540-559 176 176 176 298 298 298 F F F 28 560-579 185 185 185 312 312 312 F F F 29 580-599 185 185 185 312 312 312 F F F Max gen: 185, Max flush: 312 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.32175793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.329893351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330173747Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330200734Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330209244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330220343Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Crucible 67b59370-c3ab-465a-85ae-f95659e643fc has session id: fcd8ba71-3fcf-49e2-ad76-6620de4c122f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330556903Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Upstairs opts: Upstairs UUID: 67b59370-c3ab-465a-85ae-f95659e643fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330606279Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Crucible stats registered with UUID: 67b59370-c3ab-465a-85ae-f95659e643fc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330616858Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:47.330625717Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330769575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330801912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330812671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.33094202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330957468Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"67b59370-c3ab-465a-85ae-f95659e643fc active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330973537Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330982196Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.330991575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331001115Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331010494Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331020243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331029832Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"ds_connection connected from Ok(127.0.0.1:51607)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331094536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57763)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331105306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"ds_connection connected from Ok(127.0.0.1:41825)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331115395Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"io task","client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331834852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331884348Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.331898146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332154204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332189871Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[0]R flush_numbers[0..12]: [309, 298, 306, 312, 312, 301, 304, 295, 311, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332236957Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[0]R generation[0..12]: [182, 176, 180, 185, 185, 178, 179, 176, 184, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332248006Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332258785Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[1]R flush_numbers[0..12]: [309, 298, 306, 312, 312, 301, 304, 295, 311, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332269314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[1]R generation[0..12]: [182, 176, 180, 185, 185, 178, 179, 176, 184, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332279623Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332290292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[2]R flush_numbers[0..12]: [309, 298, 306, 312, 312, 301, 304, 295, 311, 308, 308, 310]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332300671Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[2]R generation[0..12]: [182, 176, 180, 185, 185, 178, 179, 176, 184, 181, 181, 183]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332311011Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.33232178Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332331219Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332342428Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Next flush: 313","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332351657Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332360986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"":"downstairs","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332370745Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332379425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"0","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332389054Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"1","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332404012Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"client":"2","":"downstairs","session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332413912Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332422951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"67b59370-c3ab-465a-85ae-f95659e643fc is now active with session: fcd8ba71-3fcf-49e2-ad76-6620de4c122f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.33243272Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332441329Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"session_id":"fcd8ba71-3fcf-49e2-ad76-6620de4c122f"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332450228Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Activated sub_volume 67b59370-c3ab-465a-85ae-f95659e643fc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332458108Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332466197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.332814697Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} 01/30 Write block 178 len 4096 data: 10 02/30 Write block 226 len 16384 data: 12 13 16 16 03/30 Write block 39 len 32768 data: 7 6 6 7 9 14 13 16 04/30 Write block 285 len 40960 data: 12 12 12 12 10 10 11 11 14 12 05/30 Read block 328 len 40960 06/30 Write block 116 len 24576 data: 12 10 12 12 13 11 07/30 Write block 490 len 36864 data: 11 9 9 10 8 9 10 11 13 08/30 Write block 542 len 36864 data: 8 9 8 9 9 9 9 9 9 09/30 Write block 128 len 20480 data: 19 17 18 20 18 10/30 Read block 20 len 32768 11/30 Write block 593 len 20480 data: 15 13 13 14 12 12/30 Write block 128 len 32768 data: 20 18 19 21 19 16 16 17 13/30 Write block 483 len 8192 data: 18 16 14/30 Write block 2 len 8192 data: 8 9 15/30 Flush 16/30 Write block 492 len 36864 data: 10 11 9 10 11 12 14 14 9 17/30 Read block 333 len 36864 18/30 Read block 88 len 8192 19/30 Read block 260 len 32768 20/30 Write block 5 len 36864 data: 9 11 14 16 16 13 15 13 14 21/30 Read block 362 len 32768 22/30 Read block 297 len 36864 23/30 Read block 89 len 24576 24/30 Write block 87 len 20480 data: 16 22 22 20 17 25/30 Write block 286 len 24576 data: 13 13 13 11 11 12 26/30 Write block 113 len 36864 data: 13 12 12 13 11 13 13 14 12 27/30 Read block 291 len 32768 28/30 Read block 556 len 24576 29/30 Write block 199 len 20480 data: 9 8 8 8 7 30/30 Read block 195 len 24576 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Write 4 Done Done Sent false 1002 Acked Write 8 Done Done Sent false 1003 Acked Write 10 Done Done Sent false 1004 Acked Read 10 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 5 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:47.341091376Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:47.34127382Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1883,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 186 186 182 313 313 309 T T F <--- 1 020-039 186 186 176 313 313 298 F F F <--- 2 040-059 186 186 180 313 313 306 F F F <--- 3 060-079 185 185 185 312 312 312 F F F 4 080-099 185 185 185 312 312 312 T T F <--- 5 100-119 186 186 178 313 313 301 T T F <--- 6 120-139 186 186 179 313 313 304 T T F <--- 7 140-159 176 176 176 295 295 295 F F F 8 160-179 186 186 184 313 313 311 F F F <--- 9 180-199 181 181 181 308 308 308 T T F <--- 10 200-219 181 181 181 308 308 308 T T F <--- 11 220-239 186 186 183 313 313 310 F F F <--- 12 240-259 180 180 180 307 307 307 F F F 13 260-279 185 185 185 312 312 312 F F F 14 280-299 186 186 183 313 313 310 T T F <--- 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 185 185 185 312 312 312 F F F 19 380-399 185 185 185 312 312 312 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 183 183 310 310 310 F F F 23 460-479 185 185 185 312 312 312 F F F 24 480-499 186 186 183 313 313 310 T T F <--- 25 500-519 185 185 185 312 312 312 T T F <--- 26 520-539 181 181 181 308 308 308 F F F 27 540-559 186 186 176 313 313 298 F F F <--- 28 560-579 185 185 185 312 312 312 F F F 29 580-599 186 186 185 313 313 312 F F F <--- Max gen: 186, Max flush: 313 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 186 186 186 313 313 313 F F F 2 040-059 186 186 186 313 313 313 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 187 187 187 314 314 314 F F F 5 100-119 187 187 187 314 314 314 F F F 6 120-139 187 187 187 314 314 314 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 187 187 187 314 314 314 F F F 10 200-219 187 187 187 314 314 314 F F F 11 220-239 186 186 186 313 313 313 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 185 185 185 312 312 312 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 185 185 185 312 312 312 F F F 19 380-399 185 185 185 312 312 312 F F F 20 400-419 174 174 174 292 292 292 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 183 183 310 310 310 F F F 23 460-479 185 185 185 312 312 312 F F F 24 480-499 187 187 187 314 314 314 F F F 25 500-519 187 187 187 314 314 314 F F F 26 520-539 181 181 181 308 308 308 F F F 27 540-559 186 186 186 313 313 313 F F F 28 560-579 185 185 185 312 312 312 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 187, Max flush: 314 Begin loop 94 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.603231224Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:51.610445696Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.610499701Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.617468754Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.617723412Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.617748319Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.617756499Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.617767778Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Crucible e05d5d61-f50a-4afd-a4d0-88adc69c7297 has session id: cc33c795-8d0a-4515-9754-d41cb33ba8f4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618167323Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Upstairs opts: Upstairs UUID: e05d5d61-f50a-4afd-a4d0-88adc69c7297, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.6181965Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Crucible stats registered with UUID: e05d5d61-f50a-4afd-a4d0-88adc69c7297","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618207519Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:51.618216759Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618347797Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618364146Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618374125Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618530921Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.61854743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"e05d5d61-f50a-4afd-a4d0-88adc69c7297 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618570938Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618580337Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618589916Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618599305Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618608624Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618618194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618637602Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:39688)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618686678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56001)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618697397Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:60998)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.618707456Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"io task","client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619355119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619382917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619498817Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619655103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619681481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[0]R flush_numbers[0..12]: [314, 313, 313, 312, 314, 314, 314, 295, 313, 314, 314, 313]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619773203Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[0]R generation[0..12]: [187, 186, 186, 185, 187, 187, 187, 176, 186, 187, 187, 186]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619785092Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619795921Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[1]R flush_numbers[0..12]: [314, 313, 313, 312, 314, 314, 314, 295, 313, 314, 314, 313]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.61980639Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[1]R generation[0..12]: [187, 186, 186, 185, 187, 187, 187, 176, 186, 187, 187, 186]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619817069Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619827798Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[2]R flush_numbers[0..12]: [314, 313, 313, 312, 314, 314, 314, 295, 313, 314, 314, 313]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619838227Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[2]R generation[0..12]: [187, 186, 186, 185, 187, 187, 187, 176, 186, 187, 187, 186]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619848606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619859296Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619868425Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619877964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Next flush: 315","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619886933Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619896122Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"":"downstairs","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619906881Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619915301Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"0","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.61992481Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"1","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619934259Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"client":"2","":"downstairs","session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619944358Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619953457Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"e05d5d61-f50a-4afd-a4d0-88adc69c7297 is now active with session: cc33c795-8d0a-4515-9754-d41cb33ba8f4","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619963246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619971826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"session_id":"cc33c795-8d0a-4515-9754-d41cb33ba8f4"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619980855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Activated sub_volume e05d5d61-f50a-4afd-a4d0-88adc69c7297","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.619988744Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.620013342Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.620349063Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} 01/30 Write block 483 len 16384 data: 19 17 13 14 02/30 Write block 270 len 28672 data: 17 17 17 17 16 12 12 03/30 Write block 416 len 4096 data: 18 04/30 Write block 407 len 12288 data: 21 19 15 05/30 Write block 444 len 8192 data: 19 16 06/30 Write block 271 len 8192 data: 18 18 07/30 Write block 386 len 8192 data: 19 24 08/30 Read block 160 len 32768 09/30 Read block 390 len 8192 10/30 Read block 377 len 8192 11/30 Read block 377 len 20480 12/30 Read block 15 len 20480 13/30 Write block 232 len 28672 data: 19 18 17 20 17 18 18 14/30 Write block 376 len 16384 data: 18 19 17 17 15/30 Read block 187 len 36864 16/30 Read block 139 len 32768 17/30 Read block 361 len 20480 18/30 Read block 526 len 20480 19/30 Read block 579 len 28672 20/30 Read block 447 len 36864 21/30 Write block 548 len 4096 data: 10 22/30 Write block 27 len 28672 data: 12 13 9 9 11 11 13 23/30 Flush 24/30 Flush 25/30 Write block 106 len 24576 data: 13 14 12 9 11 10 26/30 Flush 27/30 Write block 205 len 24576 data: 9 9 10 9 9 10 28/30 Write block 482 len 16384 data: 20 20 18 14 29/30 Write block 518 len 12288 data: 19 20 19 30/30 Read block 3 len 8192 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Write 7 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 2 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Write 4 Sent Sent Done false 1028 Acked Write 3 Sent Sent Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 0 33 Done 0 27 30 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:51.627480172Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:51.627642997Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1891,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 186 188 188 313 315 315 F F F <--- 2 040-059 186 186 186 313 313 313 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 187 187 187 314 314 314 F F F 5 100-119 187 188 188 314 317 317 F F F <--- 6 120-139 187 187 187 314 314 314 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 187 187 187 314 314 314 F F F 10 200-219 187 187 187 314 314 314 F T T <--- 11 220-239 186 188 188 313 315 315 F F F <--- 12 240-259 180 180 180 307 307 307 F F F 13 260-279 185 188 188 312 315 315 F F F <--- 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 185 188 188 312 315 315 F F F <--- 19 380-399 185 188 188 312 315 315 F F F <--- 20 400-419 174 188 188 292 315 315 F F F <--- 21 420-439 183 183 183 310 310 310 F F F 22 440-459 183 188 188 310 315 315 F F F <--- 23 460-479 185 185 185 312 312 312 F F F 24 480-499 187 188 188 314 315 315 F T T <--- 25 500-519 187 187 187 314 314 314 F T T <--- 26 520-539 181 181 181 308 308 308 F T T <--- 27 540-559 186 188 188 313 315 315 F F F <--- 28 560-579 185 185 185 312 312 312 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 188, Max flush: 317 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 188 188 188 315 315 315 F F F 2 040-059 186 186 186 313 313 313 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 187 187 187 314 314 314 F F F 5 100-119 188 188 188 317 317 317 F F F 6 120-139 187 187 187 314 314 314 F F F 7 140-159 176 176 176 295 295 295 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 187 187 187 314 314 314 F F F 10 200-219 189 189 189 318 318 318 F F F 11 220-239 188 188 188 315 315 315 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 188 188 188 315 315 315 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 F F F 17 340-359 182 182 182 309 309 309 F F F 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 188 188 188 315 315 315 F F F 21 420-439 183 183 183 310 310 310 F F F 22 440-459 188 188 188 315 315 315 F F F 23 460-479 185 185 185 312 312 312 F F F 24 480-499 189 189 189 318 318 318 F F F 25 500-519 189 189 189 318 318 318 F F F 26 520-539 189 189 189 318 318 318 F F F 27 540-559 188 188 188 315 315 315 F F F 28 560-579 185 185 185 312 312 312 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 189, Max flush: 318 Begin loop 95 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.888915552Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897227578Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897502584Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897528072Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897539521Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89755095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Crucible b21caa82-a51c-48ae-8fbe-b60ec5dbadae has session id: 5122ca67-3d94-4c55-a153-0423ac612392","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897918598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Upstairs opts: Upstairs UUID: b21caa82-a51c-48ae-8fbe-b60ec5dbadae, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897949675Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Crucible stats registered with UUID: b21caa82-a51c-48ae-8fbe-b60ec5dbadae","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.897960274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:55.897969354Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898099482Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898115491Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89812546Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898283236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898298815Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"b21caa82-a51c-48ae-8fbe-b60ec5dbadae active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898333792Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898343041Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89835259Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898362119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898371598Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898382648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898410485Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"ds_connection connected from Ok(127.0.0.1:59102)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898457931Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"ds_connection connected from Ok(127.0.0.1:57342)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89846899Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"ds_connection connected from Ok(127.0.0.1:47992)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.898486878Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"io task","client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899217335Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899242013Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899253492Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899558415Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899568244Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[0]R flush_numbers[0..12]: [314, 315, 313, 312, 314, 317, 314, 295, 313, 314, 318, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899603761Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[0]R generation[0..12]: [187, 188, 186, 185, 187, 188, 187, 176, 186, 187, 189, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89961488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899631389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[1]R flush_numbers[0..12]: [314, 315, 313, 312, 314, 317, 314, 295, 313, 314, 318, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899642008Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[1]R generation[0..12]: [187, 188, 186, 185, 187, 188, 187, 176, 186, 187, 189, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899654207Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899664786Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[2]R flush_numbers[0..12]: [314, 315, 313, 312, 314, 317, 314, 295, 313, 314, 318, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899675165Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[2]R generation[0..12]: [187, 188, 186, 185, 187, 188, 187, 176, 186, 187, 189, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899685414Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899695853Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899704872Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899714271Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Next flush: 319","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899723311Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.89973243Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"":"downstairs","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899741989Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899750418Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"0","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899759898Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"1","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899769097Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"client":"2","":"downstairs","session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899778646Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899787455Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"b21caa82-a51c-48ae-8fbe-b60ec5dbadae is now active with session: 5122ca67-3d94-4c55-a153-0423ac612392","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899798184Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899806713Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"session_id":"5122ca67-3d94-4c55-a153-0423ac612392"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899815563Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Activated sub_volume b21caa82-a51c-48ae-8fbe-b60ec5dbadae","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899823352Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.899831301Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.90018836Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} 01/30 Write block 467 len 28672 data: 13 12 14 14 11 12 14 02/30 Read block 430 len 12288 03/30 Read block 234 len 24576 04/30 Read block 495 len 16384 05/30 Read block 393 len 32768 06/30 Read block 14 len 4096 07/30 Read block 3 len 12288 08/30 Flush 09/30 Write block 355 len 40960 data: 11 9 9 11 11 11 16 17 16 14 10/30 Write block 567 len 40960 data: 10 11 15 14 15 13 11 12 12 14 11/30 Write block 57 len 12288 data: 11 12 13 12/30 Write block 493 len 4096 data: 12 13/30 Read block 202 len 24576 14/30 Read block 83 len 28672 15/30 Write block 150 len 32768 data: 13 14 11 17 18 18 16 18 16/30 Flush 17/30 Write block 568 len 32768 data: 12 16 15 16 14 12 13 13 18/30 Read block 498 len 4096 19/30 Flush 20/30 Write block 142 len 28672 data: 10 10 12 13 12 11 10 21/30 Read block 462 len 28672 22/30 Write block 341 len 20480 data: 17 16 14 13 14 23/30 Write block 206 len 16384 data: 10 11 10 10 24/30 Write block 356 len 40960 data: 10 10 12 12 12 17 18 17 15 15 25/30 Flush 26/30 Write block 326 len 40960 data: 17 16 18 16 16 15 13 15 16 14 27/30 Read block 541 len 24576 28/30 Write block 588 len 32768 data: 17 18 18 19 17 16 14 14 29/30 Write block 590 len 20480 data: 19 20 18 17 15 30/30 Write block 414 len 28672 data: 16 14 19 16 18 16 17 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 28 3 34 Done 27 2 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:34:55.909043409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:34:55.909220523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1899,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 188 188 188 315 315 315 F F F 2 040-059 190 186 190 320 313 320 F F F <--- 3 060-079 185 185 185 312 312 312 F F F 4 080-099 187 187 187 314 314 314 F F F 5 100-119 188 188 188 317 317 317 F F F 6 120-139 187 187 187 314 314 314 F F F 7 140-159 190 176 190 322 295 322 F F F <--- 8 160-179 186 186 186 313 313 313 F F F 9 180-199 187 187 187 314 314 314 F F F 10 200-219 190 189 190 322 318 322 F F F <--- 11 220-239 188 188 188 315 315 315 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 188 188 188 315 315 315 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 183 183 183 310 310 310 T F T <--- 17 340-359 190 182 190 322 309 322 F F F <--- 18 360-379 190 188 190 322 315 322 F F F <--- 19 380-399 188 188 188 315 315 315 F F F 20 400-419 188 188 188 315 315 315 T F T <--- 21 420-439 183 183 183 310 310 310 T F T <--- 22 440-459 188 188 188 315 315 315 F F F 23 460-479 190 185 190 319 312 319 F T F <--- 24 480-499 190 189 190 320 318 320 F F F <--- 25 500-519 189 189 189 318 318 318 F F F 26 520-539 189 189 189 318 318 318 F F F 27 540-559 188 188 188 315 315 315 F F F 28 560-579 190 185 190 321 312 321 F F F <--- 29 580-599 186 186 186 313 313 313 T F T <--- Max gen: 190, Max flush: 322 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 188 188 188 315 315 315 F F F 2 040-059 190 190 190 320 320 320 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 187 187 187 314 314 314 F F F 5 100-119 188 188 188 317 317 317 F F F 6 120-139 187 187 187 314 314 314 F F F 7 140-159 190 190 190 322 322 322 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 187 187 187 314 314 314 F F F 10 200-219 190 190 190 322 322 322 F F F 11 220-239 188 188 188 315 315 315 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 188 188 188 315 315 315 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 191 191 191 323 323 323 F F F 17 340-359 190 190 190 322 322 322 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 191 191 191 323 323 323 F F F 21 420-439 191 191 191 323 323 323 F F F 22 440-459 188 188 188 315 315 315 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 190 190 190 320 320 320 F F F 25 500-519 189 189 189 318 318 318 F F F 26 520-539 189 189 189 318 318 318 F F F 27 540-559 188 188 188 315 315 315 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 191, Max flush: 323 Begin loop 96 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.160823561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.168962032Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169239918Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169269995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169278554Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169290043Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Crucible a9e6237e-3c75-4184-aa62-828951c714d7 has session id: f26727c2-5c15-4300-bbc6-036ec9a56afc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169650632Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Upstairs opts: Upstairs UUID: a9e6237e-3c75-4184-aa62-828951c714d7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169681119Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Crucible stats registered with UUID: a9e6237e-3c75-4184-aa62-828951c714d7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169694408Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:00.169703777Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169824327Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169840345Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.169850225Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170004821Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.1700173Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"a9e6237e-3c75-4184-aa62-828951c714d7 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170035458Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170044308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170053977Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170063406Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170072755Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170082464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170092103Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"ds_connection connected from Ok(127.0.0.1:47788)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170167937Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"ds_connection connected from Ok(127.0.0.1:44022)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170186935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"ds_connection connected from Ok(127.0.0.1:37188)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170196944Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"io task","client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170777774Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170833389Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.170875095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.17105478Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171112605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[0]R flush_numbers[0..12]: [314, 315, 320, 312, 314, 317, 314, 322, 313, 314, 322, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171156201Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[0]R generation[0..12]: [187, 188, 190, 185, 187, 188, 187, 190, 186, 187, 190, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.1711672Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171178199Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[1]R flush_numbers[0..12]: [314, 315, 320, 312, 314, 317, 314, 322, 313, 314, 322, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171188698Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[1]R generation[0..12]: [187, 188, 190, 185, 187, 188, 187, 190, 186, 187, 190, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171199027Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171209716Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[2]R flush_numbers[0..12]: [314, 315, 320, 312, 314, 317, 314, 322, 313, 314, 322, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171220145Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[2]R generation[0..12]: [187, 188, 190, 185, 187, 188, 187, 190, 186, 187, 190, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171230464Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171248743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171258062Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171267561Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Next flush: 324","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.17127674Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171285969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"":"downstairs","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171295409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171304028Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"0","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171313467Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"1","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171322686Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"client":"2","":"downstairs","session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171332215Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171341095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"a9e6237e-3c75-4184-aa62-828951c714d7 is now active with session: f26727c2-5c15-4300-bbc6-036ec9a56afc","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171350864Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171359293Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"session_id":"f26727c2-5c15-4300-bbc6-036ec9a56afc"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171368142Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Activated sub_volume a9e6237e-3c75-4184-aa62-828951c714d7","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171375852Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171383771Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.171716992Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} 01/30 Write block 80 len 12288 data: 13 14 11 02/30 Read block 257 len 32768 03/30 Read block 43 len 32768 04/30 Read block 231 len 12288 05/30 Write block 185 len 28672 data: 10 8 9 7 7 11 15 06/30 Flush 07/30 Write block 80 len 32768 data: 14 15 12 12 13 16 18 17 08/30 Read block 79 len 24576 09/30 Write block 391 len 20480 data: 22 28 22 26 22 10/30 Write block 201 len 8192 data: 9 9 11/30 Write block 104 len 40960 data: 16 17 14 15 13 10 12 11 9 14 12/30 Read block 413 len 24576 13/30 Read block 186 len 36864 14/30 Read block 360 len 32768 15/30 Write block 542 len 40960 data: 9 10 9 10 10 10 11 10 10 9 16/30 Read block 168 len 4096 17/30 Read block 186 len 16384 18/30 Flush 19/30 Write block 527 len 8192 data: 12 11 20/30 Read block 80 len 24576 21/30 Write block 480 len 28672 data: 16 18 21 21 19 15 15 22/30 Write block 29 len 12288 data: 10 10 12 23/30 Write block 399 len 8192 data: 21 24 24/30 Flush 25/30 Write block 454 len 24576 data: 15 16 14 11 15 13 26/30 Write block 552 len 32768 data: 10 10 13 16 14 13 13 13 27/30 Read block 246 len 20480 28/30 Write block 132 len 28672 data: 20 17 17 18 18 16 12 29/30 Write block 457 len 4096 data: 12 30/30 Read block 21 len 20480 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Read 8 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Write 1 Sent Sent Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 0 32 Done 28 0 30 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:00.180536324Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:00.180728197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1907,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 192 188 192 326 315 326 F F F <--- 2 040-059 190 190 190 320 320 320 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 192 187 192 325 314 325 F F F <--- 5 100-119 192 188 192 325 317 325 F F F <--- 6 120-139 187 187 187 314 314 314 T F T <--- 7 140-159 190 190 190 322 322 322 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 192 187 192 324 314 324 F F F <--- 10 200-219 192 190 192 325 322 325 F F F <--- 11 220-239 188 188 188 315 315 315 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 188 188 188 315 315 315 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 191 191 191 323 323 323 F F F 17 340-359 190 190 190 322 322 322 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 192 188 192 326 315 326 F F F <--- 20 400-419 192 191 192 326 323 326 F F F <--- 21 420-439 191 191 191 323 323 323 F F F 22 440-459 188 188 188 315 315 315 T F T <--- 23 460-479 190 190 190 319 319 319 F F F 24 480-499 192 190 192 326 320 326 F F F <--- 25 500-519 189 189 189 318 318 318 F F F 26 520-539 192 189 192 326 318 326 F F F <--- 27 540-559 192 188 192 325 315 325 T F T <--- 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 192, Max flush: 326 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 192 192 192 326 326 326 F F F 2 040-059 190 190 190 320 320 320 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 192 192 192 325 325 325 F F F 5 100-119 192 192 192 325 325 325 F F F 6 120-139 193 193 193 327 327 327 F F F 7 140-159 190 190 190 322 322 322 F F F 8 160-179 186 186 186 313 313 313 F F F 9 180-199 192 192 192 324 324 324 F F F 10 200-219 192 192 192 325 325 325 F F F 11 220-239 188 188 188 315 315 315 F F F 12 240-259 180 180 180 307 307 307 F F F 13 260-279 188 188 188 315 315 315 F F F 14 280-299 187 187 187 314 314 314 F F F 15 300-319 183 183 183 310 310 310 F F F 16 320-339 191 191 191 323 323 323 F F F 17 340-359 190 190 190 322 322 322 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 192 192 192 326 326 326 F F F 20 400-419 192 192 192 326 326 326 F F F 21 420-439 191 191 191 323 323 323 F F F 22 440-459 193 193 193 327 327 327 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 192 192 192 326 326 326 F F F 25 500-519 189 189 189 318 318 318 F F F 26 520-539 192 192 192 326 326 326 F F F 27 540-559 193 193 193 327 327 327 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 193, Max flush: 327 Begin loop 97 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.433431419Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.441548461Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.441833897Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.441857475Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.441865804Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.441877043Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Crucible 7e4b65a3-b4c9-4469-bf2b-94e22584b844 has session id: 5034f0f3-9518-49dd-9513-282e066c00f6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442223073Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Upstairs opts: Upstairs UUID: 7e4b65a3-b4c9-4469-bf2b-94e22584b844, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442245321Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Crucible stats registered with UUID: 7e4b65a3-b4c9-4469-bf2b-94e22584b844","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44225523Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:04.442264369Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442432064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442461292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442472031Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442583731Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44259927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"7e4b65a3-b4c9-4469-bf2b-94e22584b844 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442634917Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442644236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442653935Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442663444Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442672793Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442682493Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442692192Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"ds_connection connected from Ok(127.0.0.1:34785)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442753976Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"ds_connection connected from Ok(127.0.0.1:38564)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.442805002Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"ds_connection connected from Ok(127.0.0.1:36625)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44282438Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"io task","client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443452236Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443493762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44351839Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443760409Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443780477Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[0]R flush_numbers[0..12]: [314, 326, 320, 312, 325, 325, 327, 322, 313, 324, 325, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443840362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[0]R generation[0..12]: [187, 192, 190, 185, 192, 192, 193, 190, 186, 192, 192, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443851441Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44386217Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[1]R flush_numbers[0..12]: [314, 326, 320, 312, 325, 325, 327, 322, 313, 324, 325, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443872619Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[1]R generation[0..12]: [187, 192, 190, 185, 192, 192, 193, 190, 186, 192, 192, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443882828Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443893457Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[2]R flush_numbers[0..12]: [314, 326, 320, 312, 325, 325, 327, 322, 313, 324, 325, 315]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443903826Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[2]R generation[0..12]: [187, 192, 190, 185, 192, 192, 193, 190, 186, 192, 192, 188]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443914095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443924784Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443934004Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443943543Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Next flush: 328","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443952732Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443961951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"":"downstairs","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.44397308Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443981629Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"0","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.443991019Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"1","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444000488Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"client":"2","":"downstairs","session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444010147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444019096Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"7e4b65a3-b4c9-4469-bf2b-94e22584b844 is now active with session: 5034f0f3-9518-49dd-9513-282e066c00f6","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444028885Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444037555Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"session_id":"5034f0f3-9518-49dd-9513-282e066c00f6"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444046384Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Activated sub_volume 7e4b65a3-b4c9-4469-bf2b-94e22584b844","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444054253Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444062292Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.444360486Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} 01/30 Write block 494 len 32768 data: 10 11 12 13 15 15 10 9 02/30 Read block 532 len 36864 03/30 Read block 233 len 24576 04/30 Read block 15 len 36864 05/30 Read block 147 len 32768 06/30 Write block 276 len 28672 data: 13 13 15 14 15 15 13 07/30 Read block 286 len 12288 08/30 Read block 104 len 28672 09/30 Read block 304 len 20480 10/30 Flush 11/30 Write block 320 len 24576 data: 20 15 17 19 18 18 12/30 Write block 294 len 24576 data: 13 16 17 16 16 21 13/30 Read block 291 len 8192 14/30 Write block 159 len 16384 data: 15 15 13 13 15/30 Write block 547 len 4096 data: 11 16/30 Read block 464 len 16384 17/30 Read block 392 len 24576 18/30 Write block 236 len 40960 data: 18 19 19 15 13 10 10 10 10 10 19/30 Write block 226 len 12288 data: 13 14 17 20/30 Flush 21/30 Write block 264 len 24576 data: 17 21 21 17 16 16 22/30 Write block 391 len 28672 data: 23 29 23 27 23 17 19 23/30 Write block 303 len 8192 data: 23 23 24/30 Write block 25 len 12288 data: 13 15 13 25/30 Write block 513 len 24576 data: 20 19 18 20 21 20 26/30 Write block 323 len 24576 data: 20 19 19 18 17 19 27/30 Read block 366 len 40960 28/30 Write block 112 len 32768 data: 10 15 13 13 14 12 14 14 29/30 Read block 173 len 28672 30/30 Read block 12 len 24576 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 30 2 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:04.453307187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:04.45350064Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1915,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 192 192 192 326 326 326 T F T <--- 2 040-059 190 190 190 320 320 320 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 192 192 192 325 325 325 F F F 5 100-119 192 192 192 325 325 325 T F T <--- 6 120-139 193 193 193 327 327 327 F F F 7 140-159 194 190 194 329 322 329 F F F <--- 8 160-179 194 186 194 329 313 329 F F F <--- 9 180-199 192 192 192 324 324 324 F F F 10 200-219 192 192 192 325 325 325 F F F 11 220-239 194 188 194 329 315 329 F F F <--- 12 240-259 194 180 194 329 307 329 F F F <--- 13 260-279 194 188 194 328 315 328 T F T <--- 14 280-299 194 187 194 329 314 329 F F F <--- 15 300-319 183 183 183 310 310 310 T F T <--- 16 320-339 194 191 194 329 323 329 T F T <--- 17 340-359 190 190 190 322 322 322 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 192 192 192 326 326 326 T F T <--- 20 400-419 192 192 192 326 326 326 F F F 21 420-439 191 191 191 323 323 323 F F F 22 440-459 193 193 193 327 327 327 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 192 194 328 326 328 F T F <--- 25 500-519 194 189 194 328 318 328 T T T <--- 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 193 194 329 327 329 F F F <--- 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 194, Max flush: 329 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 195 195 195 330 330 330 F F F 2 040-059 190 190 190 320 320 320 F F F 3 060-079 185 185 185 312 312 312 F F F 4 080-099 192 192 192 325 325 325 F F F 5 100-119 195 195 195 330 330 330 F F F 6 120-139 193 193 193 327 327 327 F F F 7 140-159 194 194 194 329 329 329 F F F 8 160-179 194 194 194 329 329 329 F F F 9 180-199 192 192 192 324 324 324 F F F 10 200-219 192 192 192 325 325 325 F F F 11 220-239 194 194 194 329 329 329 F F F 12 240-259 194 194 194 329 329 329 F F F 13 260-279 195 195 195 330 330 330 F F F 14 280-299 194 194 194 329 329 329 F F F 15 300-319 195 195 195 330 330 330 F F F 16 320-339 195 195 195 330 330 330 F F F 17 340-359 190 190 190 322 322 322 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 195 195 195 330 330 330 F F F 20 400-419 192 192 192 326 326 326 F F F 21 420-439 191 191 191 323 323 323 F F F 22 440-459 193 193 193 327 327 327 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 194 194 328 328 328 F F F 25 500-519 195 195 195 330 330 330 F F F 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 194 194 329 329 329 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 195, Max flush: 330 Begin loop 98 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.711769267Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:08.719003437Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.719074971Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72619642Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726459707Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726503353Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726513762Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726525111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Crucible c0fdcb53-19c8-4115-8028-7b0f1fa89aee has session id: c4ac145f-2b99-4b09-b066-5cf6d0f29ba5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726840844Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Upstairs opts: Upstairs UUID: c0fdcb53-19c8-4115-8028-7b0f1fa89aee, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.726879981Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Crucible stats registered with UUID: c0fdcb53-19c8-4115-8028-7b0f1fa89aee","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72689018Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:08.726899089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727023008Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727039187Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727048966Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727198143Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727213502Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"c0fdcb53-19c8-4115-8028-7b0f1fa89aee active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72722963Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727238429Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727249208Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727258558Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727267927Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727277676Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727287575Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49696)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727382197Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37300)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727407245Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"ds_connection connected from Ok(127.0.0.1:39782)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.727418314Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"io task","client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728010712Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728024391Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728111423Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728309046Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728326855Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[0]R flush_numbers[0..12]: [314, 330, 320, 312, 325, 330, 327, 329, 329, 324, 325, 329]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728400838Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[0]R generation[0..12]: [187, 195, 190, 185, 192, 195, 193, 194, 194, 192, 192, 194]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728412147Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728425066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[1]R flush_numbers[0..12]: [314, 330, 320, 312, 325, 330, 327, 329, 329, 324, 325, 329]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728435605Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[1]R generation[0..12]: [187, 195, 190, 185, 192, 195, 193, 194, 194, 192, 192, 194]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728445964Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728456843Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[2]R flush_numbers[0..12]: [314, 330, 320, 312, 325, 330, 327, 329, 329, 324, 325, 329]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728467202Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[2]R generation[0..12]: [187, 195, 190, 185, 192, 195, 193, 194, 194, 192, 192, 194]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728477341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72848801Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72849739Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728506969Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728516268Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728525427Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"":"downstairs","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728534986Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728543606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"0","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728553035Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"1","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728562504Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"client":"2","":"downstairs","session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728573453Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728582312Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"c0fdcb53-19c8-4115-8028-7b0f1fa89aee is now active with session: c4ac145f-2b99-4b09-b066-5cf6d0f29ba5","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728592091Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728600681Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"session_id":"c4ac145f-2b99-4b09-b066-5cf6d0f29ba5"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.72860943Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Activated sub_volume c0fdcb53-19c8-4115-8028-7b0f1fa89aee","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728617179Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728625089Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.728911274Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} 01/30 Write block 355 len 4096 data: 12 02/30 Read block 168 len 4096 03/30 Write block 198 len 8192 data: 11 10 04/30 Read block 245 len 12288 05/30 Write block 98 len 16384 data: 9 11 11 14 06/30 Flush 07/30 Write block 72 len 28672 data: 14 10 13 14 11 12 12 08/30 Write block 130 len 12288 data: 20 22 21 09/30 Read block 152 len 32768 10/30 Write block 263 len 28672 data: 17 18 22 22 18 17 17 11/30 Read block 82 len 40960 12/30 Read block 69 len 12288 13/30 Read block 77 len 40960 14/30 Write block 439 len 16384 data: 17 18 19 20 15/30 Flush 16/30 Write block 210 len 28672 data: 11 14 15 14 12 11 11 17/30 Read block 532 len 24576 18/30 Write block 107 len 20480 data: 16 14 11 13 12 19/30 Write block 508 len 40960 data: 19 16 14 14 17 21 20 19 21 22 20/30 Read block 307 len 4096 21/30 Read block 80 len 20480 22/30 Read block 82 len 4096 23/30 Read block 212 len 12288 24/30 Read block 1 len 24576 25/30 Write block 100 len 40960 data: 12 15 14 15 17 18 15 17 15 12 26/30 Write block 37 len 28672 data: 11 9 8 7 7 8 10 27/30 Read block 401 len 24576 28/30 Write block 212 len 16384 data: 16 15 13 12 29/30 Read block 136 len 40960 30/30 Write block 237 len 36864 data: 20 20 16 14 11 11 11 11 11 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 7 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:08.737929648Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:08.738093124Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1923,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 195 195 195 330 330 330 F T T <--- 2 040-059 190 190 190 320 320 320 F T T <--- 3 060-079 185 196 196 312 332 332 F F F <--- 4 080-099 192 196 196 325 331 331 T F F <--- 5 100-119 195 196 196 330 331 331 T T T <--- 6 120-139 193 196 196 327 332 332 F F F <--- 7 140-159 194 194 194 329 329 329 F F F 8 160-179 194 194 194 329 329 329 F F F 9 180-199 192 196 196 324 331 331 T F F <--- 10 200-219 192 192 192 325 325 325 F T T <--- 11 220-239 194 194 194 329 329 329 F T T <--- 12 240-259 194 194 194 329 329 329 F T T <--- 13 260-279 195 196 196 330 332 332 F F F <--- 14 280-299 194 194 194 329 329 329 F F F 15 300-319 195 195 195 330 330 330 F F F 16 320-339 195 195 195 330 330 330 F F F 17 340-359 190 196 196 322 331 331 T F F <--- 18 360-379 190 190 190 322 322 322 F F F 19 380-399 195 195 195 330 330 330 F F F 20 400-419 192 192 192 326 326 326 F F F 21 420-439 191 196 196 323 332 332 F F F <--- 22 440-459 193 196 196 327 332 332 F F F <--- 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 194 194 328 328 328 F F F 25 500-519 195 195 195 330 330 330 F T T <--- 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 194 194 329 329 329 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 196, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 197 197 197 333 333 333 F F F 2 040-059 197 197 197 333 333 333 F F F 3 060-079 196 196 196 332 332 332 F F F 4 080-099 196 196 196 331 331 331 F F F 5 100-119 197 197 197 333 333 333 F F F 6 120-139 196 196 196 332 332 332 F F F 7 140-159 194 194 194 329 329 329 F F F 8 160-179 194 194 194 329 329 329 F F F 9 180-199 196 196 196 331 331 331 F F F 10 200-219 197 197 197 333 333 333 F F F 11 220-239 197 197 197 333 333 333 F F F 12 240-259 197 197 197 333 333 333 F F F 13 260-279 196 196 196 332 332 332 F F F 14 280-299 194 194 194 329 329 329 F F F 15 300-319 195 195 195 330 330 330 F F F 16 320-339 195 195 195 330 330 330 F F F 17 340-359 196 196 196 331 331 331 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 195 195 195 330 330 330 F F F 20 400-419 192 192 192 326 326 326 F F F 21 420-439 196 196 196 332 332 332 F F F 22 440-459 196 196 196 332 332 332 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 194 194 328 328 328 F F F 25 500-519 197 197 197 333 333 333 F F F 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 194 194 329 329 329 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 F F F Max gen: 197, Max flush: 333 Begin loop 99 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.004327506Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.012708226Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.012988362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"451c8887d419eecdc17bf46d23d980c26ef85fb4\",\n git_commit_timestamp: \"2026-09-18T03:11:31.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.01301351Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013022359Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013033688Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Crucible 58f3fa27-3d63-46aa-8f32-c77c56c99a00 has session id: 1f321ba2-3963-4082-b2da-455614b01f6f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013377188Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Upstairs opts: Upstairs UUID: 58f3fa27-3d63-46aa-8f32-c77c56c99a00, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013408095Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Crucible stats registered with UUID: 58f3fa27-3d63-46aa-8f32-c77c56c99a00","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013419204Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:13.013428313Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.0135773Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013607868Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013618827Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013795111Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.01381194Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"58f3fa27-3d63-46aa-8f32-c77c56c99a00 active request set","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013829988Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013838718Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013848307Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013857716Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013866945Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013877604Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013887363Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:41582)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013968796Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55516)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.013993854Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:45641)","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.014088536Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"io task","client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.014675075Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.014715451Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.01472685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.014977678Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015010736Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[0]R flush_numbers[0..12]: [314, 333, 333, 332, 331, 333, 332, 329, 329, 331, 333, 333]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015035743Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[0]R generation[0..12]: [187, 197, 197, 196, 196, 197, 196, 194, 194, 196, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015046652Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015057362Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[1]R flush_numbers[0..12]: [314, 333, 333, 332, 331, 333, 332, 329, 329, 331, 333, 333]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015067951Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[1]R generation[0..12]: [187, 197, 197, 196, 196, 197, 196, 194, 194, 196, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.01507818Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015088799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[2]R flush_numbers[0..12]: [314, 333, 333, 332, 331, 333, 332, 329, 329, 331, 333, 333]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015099068Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[2]R generation[0..12]: [187, 197, 197, 196, 196, 197, 196, 194, 194, 196, 197, 197]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015111407Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015122246Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015131495Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015141024Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015149993Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015159193Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"":"downstairs","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015168742Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015177341Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"0","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.01518677Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"1","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015196059Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"client":"2","":"downstairs","session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015205799Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015214708Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"58f3fa27-3d63-46aa-8f32-c77c56c99a00 is now active with session: 1f321ba2-3963-4082-b2da-455614b01f6f","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015224547Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015233066Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"session_id":"1f321ba2-3963-4082-b2da-455614b01f6f"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015241995Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Activated sub_volume 58f3fa27-3d63-46aa-8f32-c77c56c99a00","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015249685Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015258804Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.015585306Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} 01/30 Write block 329 len 16384 data: 17 17 16 14 02/30 Read block 74 len 16384 03/30 Read block 354 len 4096 04/30 Write block 38 len 16384 data: 10 9 8 8 05/30 Write block 293 len 16384 data: 15 14 17 18 06/30 Flush 07/30 Write block 22 len 4096 data: 10 08/30 Write block 392 len 12288 data: 30 24 28 09/30 Write block 161 len 28672 data: 14 14 11 10 11 9 7 10/30 Write block 118 len 8192 data: 15 15 11/30 Read block 318 len 8192 12/30 Read block 488 len 4096 13/30 Read block 346 len 28672 14/30 Read block 219 len 40960 15/30 Flush 16/30 Write block 302 len 36864 data: 21 24 24 21 20 19 17 16 17 17/30 Flush 18/30 Write block 280 len 12288 data: 16 16 14 19/30 Write block 124 len 20480 data: 10 11 13 14 21 20/30 Flush 21/30 Write block 505 len 36864 data: 16 20 19 20 17 15 15 18 22 22/30 Write block 454 len 4096 data: 16 23/30 Flush 24/30 Write block 160 len 40960 data: 16 15 15 12 11 12 10 8 9 9 25/30 Write block 289 len 16384 data: 12 12 13 12 26/30 Write block 588 len 28672 data: 18 19 20 21 19 18 16 27/30 Write block 305 len 12288 data: 22 21 20 28/30 Write block 515 len 16384 data: 20 22 23 21 29/30 Write block 139 len 12288 data: 13 10 10 30/30 Write block 439 len 24576 data: 18 19 20 21 16 20 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 4 Done Done Done false 1004 Acked Write 4 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 26 32 Done 27 27 4 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-09-18T03:35:13.023103421Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931} {"msg":"Wrote out file \"/tmp/test_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-09-18T03:35:13.023270606Z","hostname":"w-01M2S8T6Q0KD4MKYNHJ4YTPGK1","pid":1931,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 198 198 197 335 335 333 F F T <--- 2 040-059 198 198 197 334 334 333 F F T <--- 3 060-079 196 196 196 332 332 332 F F F 4 080-099 196 196 196 331 331 331 F F F 5 100-119 198 198 197 335 335 333 F F F <--- 6 120-139 198 198 196 337 337 332 T T F <--- 7 140-159 194 194 194 329 329 329 T T F <--- 8 160-179 198 198 194 335 335 329 T T F <--- 9 180-199 196 196 196 331 331 331 F F F 10 200-219 197 197 197 333 333 333 F F F 11 220-239 197 197 197 333 333 333 F F F 12 240-259 197 197 197 333 333 333 F F F 13 260-279 196 196 196 332 332 332 F F F 14 280-299 198 198 194 337 337 329 T T F <--- 15 300-319 198 198 195 336 336 330 T T F <--- 16 320-339 198 198 195 334 334 330 F F T <--- 17 340-359 196 196 196 331 331 331 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 198 198 195 335 335 330 F F F <--- 20 400-419 192 192 192 326 326 326 F F F 21 420-439 196 196 196 332 332 332 T T F <--- 22 440-459 198 198 196 338 338 332 T T F <--- 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 194 194 328 328 328 F F F 25 500-519 198 198 197 338 338 333 T T F <--- 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 194 194 329 329 329 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 191 191 191 323 323 323 T T F <--- Max gen: 198, Max flush: 338 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 314 314 314 F F F 1 020-039 198 198 198 335 335 335 F F F 2 040-059 198 198 198 334 334 334 F F F 3 060-079 196 196 196 332 332 332 F F F 4 080-099 196 196 196 331 331 331 F F F 5 100-119 198 198 198 335 335 335 F F F 6 120-139 199 199 199 339 339 339 F F F 7 140-159 199 199 199 339 339 339 F F F 8 160-179 199 199 199 339 339 339 F F F 9 180-199 196 196 196 331 331 331 F F F 10 200-219 197 197 197 333 333 333 F F F 11 220-239 197 197 197 333 333 333 F F F 12 240-259 197 197 197 333 333 333 F F F 13 260-279 196 196 196 332 332 332 F F F 14 280-299 199 199 199 339 339 339 F F F 15 300-319 199 199 199 339 339 339 F F F 16 320-339 198 198 198 334 334 334 F F F 17 340-359 196 196 196 331 331 331 F F F 18 360-379 190 190 190 322 322 322 F F F 19 380-399 198 198 198 335 335 335 F F F 20 400-419 192 192 192 326 326 326 F F F 21 420-439 199 199 199 339 339 339 F F F 22 440-459 199 199 199 339 339 339 F F F 23 460-479 190 190 190 319 319 319 F F F 24 480-499 194 194 194 328 328 328 F F F 25 500-519 199 199 199 339 339 339 F F F 26 520-539 192 192 192 326 326 326 F F F 27 540-559 194 194 194 329 329 329 F F F 28 560-579 190 190 190 321 321 321 F F F 29 580-599 199 199 199 339 339 339 F F F Max gen: 199, Max flush: 339 7:28 Test duration Test completed real 7:28.158884816 user 25.933128098 sys 14.430496274 trap 0.020330421 tflt 0.066548725 dflt 0.075866560 kflt 0.016576773 lock 11:57:53.183292227 slp 36:17.252345581 lat 2.417087031 stop 0.019897242